Linking Self-Perceived Cognitive Functioning Questionnaires Using Item Response Theory: The Subjective Cognitive Decline Initiative

Abstract

Objective:

Self-perceived cognitive functioning, considered highly relevant in the context of aging and dementia, is assessed in numerous ways—hindering comparison of findings across studies and settings. Therefore, the current study aimed to link item-level self-report questionnaire data from international aging studies.

Methods:

We harmonized secondary data from 24 studies and 40 different questionnaires with item response theory (IRT) techniques using a graded response model with a Bayesian estimator. We compared item information curves to identify items with high measurement precision at different levels of the self-perceived cognitive functioning latent trait. Data from 53,030 neuropsychologically intact older adults were included, from 13 English language and 11 non-English (or mixed) language studies.

Results:

We successfully linked all questionnaires and demonstrated that a single factor structure was reasonable for the latent trait. Items that made the greatest contribution to measurement precision (i.e., “top items”) assessed general and specific memory problems and aspects of executive functioning, attention, language, calculation, and visuospatial skills. These top items originated from distinct questionnaires and varied in format, range, time frames, response options, and whether they captured ability and/or change.

Conclusions:

This was the first study to calibrate self-perceived cognitive functioning data of geographically diverse older adults. The resulting item scores are on the same metric, facilitating joint or pooled analyses across international studies. Results may lead to the development of new self-perceived cognitive functioning questionnaires guided by psychometric properties, content, and other important features of items in our item bank.

Older adults commonly perceive changes in their own cognitive abilities, hereafter referred to as decline in self-perceived cognitive functioning (Jonker et al., 1996; 2000; Ponds et al., 1997; Röhr et al., 2020; van Harten et al., 2018). Although sometimes accompanied by impairment detected from assessment with objective neuropsychological tests, as in prodromal dementia conditions such as mild cognitive impairment (MCI), decline in self-perceived cognitive functioning can also be related to various medical conditions, mood or sleep disturbances, medications, personality traits, or physiological changes that occur with normal aging (Buckley et al., 2013; Comijs et al., 2002; Hill et al., 2016; Jessen et al., 2020; Rabin et al., 2017; Zlatar et al., 2014). For some older adults, a self-experienced persistent decline in cognitive functioning, in the absence of objective neuropsychological deficits (and unrelated to an acute event), may be an early manifestation of underlying neurodegenerative changes associated with an increased risk of future objective cognitive decline (Jessen et al., 2014; Jessen et al., 2020; Reisberg et al., 2008). This condition, referred to as subjective cognitive decline (SCD), may represent one of the first symptomatic manifestations of Alzheimer’s disease and related disorders (ADRD; Jessen et al., 2014; 2020; Molinuevo et al., 2017; Rabin et al., 2017; Reisberg & Gauthier, 2008; Slot et al., 2019).

The assessment of self-perceived cognitive functioning offers the advantage of directly gathering information about important and potentially clinically significant aspects of cognition and function from a first-person perspective (Lai et al., 2014; Lucas & Baird, 2006). Unfortunately, the field lacks a standardized approach for measuring self-perceived cognitive functioning in older adults, which greatly limits understanding of the construct of SCD and the ability to compare findings across studies and settings (Molinuevo et al., 2017; Röhr et al., 2020). Coincident with the conceptualization of the SCD condition as a manifestation of the late preclinical stage of Alzheimer’s disease (AD; Jack Jr. et al., 2018), there has been a proliferation of new questionnaires or adoption of existing questionnaires within international aging studies (Abdulrab & Heun, 2008; Rabin et al., 2015; Reid & MacLullich, 2006). Some measures have been adapted for application in multiple languages (Raimo et al., 2016; Şahin et al., 2013; Slot et al., 2018) and/or used internationally with culturally and linguistically diverse populations (Brucki & Nitrini, 2009; Hao et al., 2017; Vlachos et al., 2019). A previous study documented the vast heterogeneity in questionnaires used to assess self-perceived cognitive functioning in 19 aging studies from Australia and countries in North America and Europe (Rabin et al., 2015). These studies were also diverse with respect to recruitment approaches, research settings, language in which assessments were conducted, and sample sizes. The study found that while items inquired about a variety of cognitive domains, those pertaining to memory were the most common (Rabin et al., 2015). Overall, results revealed little overlap among self-perceived cognitive functioning questionnaires and great variability in terms of key features of item format and content. Additionally, questionnaire and item selection decisions were mostly based on practical rather than psychometric considerations, with many measures lacking evidence of validity (Rabin et al., 2015).

The variability in questionnaires and item properties may account, in part, for discrepant findings with regard to whether decline in self-perceived cognitive functioning is associated with AD biomarkers and clinical progression (Amariglio et al., 2012; 2015b; 2021; Buckley et al., 2016; Jessen et al., 2010; Kryscio et al., 2014; Mielke et al., 2012; Mitchell et al., 2014; Mosconi et al., 2008; Perrotin et al., 2012; Peter et al., 2014; Reisberg et al., 2010; 2019; Risacher et al., 2015; Saykin et al., 2006; Scheef et al., 2012; Schultz et al., 2015; Slot et al., 2019; Snitz et al., 2018; van Harten et al., 2013; Visser et al., 2009) or is not associated with AD biomarkers and clinical progression (Chételat et al., 2010; Hollands et al., 2015; Zwan et al., 2016; see Colijn & Grossberg, 2015; Lin et al., 2019; Lista et al., 2015; Sun et al., 2015 for reviews). Initiatives to understand the condition of SCD have led to efforts to develop a unified conceptual framework (Jessen et al., 2014; 2020). Additionally, there have been calls to pool data across studies to examine the psychometric properties and quality of items used to classify SCD (Jessen et al., 2020; Molinuevo et al., 2017, Rabin et al., 2017; Röhr et al., 2020). Investigation into the reliability and validity of self-perceived cognitive functioning measures and improvements in measurement development are critical to promoting understanding and replicability of study conclusions (Fried & Flake, 2018; Flake & Fried, 2020).

The application of psychometric modeling techniques, such as item response theory (IRT), to self-perceived cognitive functioning measurement could enable harmonization of data across studies with diverse assessment approaches. Additionally, such an approach could facilitate evaluation of the precision of items and subsequent modification of the instruments themselves. IRT comprises a collection of modeling techniques for the analysis of item level data, which can be used to evaluate the psychometric properties of existing scales and items, shorten scales, and calibrate items from different scales (Edelen & Reeve, 2007). IRT modeling can help generate precise, valid, and relatively brief instruments that present minimal burden to respondents. One example of the use of IRT in healthcare research and clinical practice is the Patient-Reported Outcomes Measurement Information System (PROMIS ^™), which consists of item banks and short forms of self-reported health measures that assess many aspects of physical, mental/cognitive, and social health in diverse adult and pediatric populations (Cella et al., 2010; Irwin et al., 2012; Northwestern University, 2022). The application of IRT to self-perceived cognitive functioning measurement has been performed in the context of oncology, thereby supporting the suitability of measuring cognitive concerns and abilities with a unidimensional item bank (Lai et al., 2014). In the field of cognitive aging, IRT has been utilized in two published studies researching item-level self-perceived cognitive functioning data. Snitz and colleagues (2011) examined the utility of IRT for items probing self-perceived cognitive functioning in 3,495 older adult participants from four community-based studies in Southwestern Pennsylvania, U.S. IRT scoring of questionnaire items was associated with objective cognitive test scores and provided additional information beyond a simple sum of items, making this approach ideal for capturing very subtle declines in self-perceived cognitive functioning. Gifford and colleagues (2015) sought to identify items that most reliably captured self-perceived cognitive functioning among 188 non-demented older adults recruited from a research registry in Boston, U.S. After post-hoc simulation using computerized adaptive testing, researchers identified nine of 21 items that represented the latent trait and differentiated normal controls from those with MCI. These items assessed self-perceived global and specific problems and changes related to memory functioning.

These previous studies supported the benefits of applying a modern psychometric approach to self-perceived cognitive functioning data. The current study expands upon this foundation by employing item banking and co-calibration to link questionnaires from international cognitive aging studies to provide a more generalizable model of self-perceived cognitive functioning. Using IRT methods, we sought to minimize measurement scale differences as a potential impediment to understanding the coherence of results across studies. We further aimed to identify items with both strong and weak psychometric characteristics based on item information functions. Harmonization results may be used to compare data from studies that use items with varied content and measurement properties and to identify optimal items to include on new self-perceived cognitive functioning questionnaires.

Methods

Figure 1 provides an overview of the 10 primary data coding and analytic processing steps, which are discussed below in the same order.

(1). Data Gathered, Coded, and Merged

Data Sources and Questionnaires

Data were obtained from members of the Subjective Cognitive Decline Initiative (SCD-I), part of the SCD Professional Interest Area of the Alzheimer’s Association International Society to Advance Alzheimer’s Research and Treatment (ISTAART SCD-PIA). The SCD-I is a working group of AD researchers with a specific interest in self-perceived cognitive functioning.

Participating researchers sent electronic copies of their published and unpublished cognitive and functional self-report questionnaires. For studies conducted in languages other than English, researchers sent translated versions of their questionnaires. For the Alzheimer’s Disease Neuroimaging Initiative (ADNI), Anti-Amyloid Treatment in Asymptomatic Alzheimer’s (A4), and Health and Aging Brain Study: Health Disparities (HABS-HD) studies, after obtaining permission we retrieved the questionnaires directly from the study databases (adni.loni.usc.edu, apps.unthsc.edu/itr/researchers, and a4study.org, respectively).

Categorization and Item Inclusion/Exclusion

Study authors reviewed all questionnaires to determine which merited inclusion by virtue of containing items related to self-perceived cognitive functioning. To make this determination, each item was assigned to one of eight primary cognitive domains: (1) memory, (2) attention/working memory/processing speed, (3) language, (4) executive function, (5) basic calculation and arithmetic tasks, (6) orientation, (7) general cognitive ability, and (8) visuospatial skills. Items that could not be assigned to a cognitive domain were excluded; such items assessed mood, health complaints, self-efficacy, perceived stress, personality, life satisfaction, basic activities of daily living, use of cognitive strategies, and other variables that are relevant to or associated with cognition but do not directly assess self-perceived cognitive functioning. Questionnaires that contained at least one self-perceived cognitive function item were retained.

For each self-perceived cognitive functioning item, beyond cognitive domain, we coded other relevant features such as the specific type(s) of response options, scaling methods, response time frames, and whether item stems were multi-barreled (i.e., containing two or more sub-questions within a single item). We also determined whether item stems and response options together inquired about the ability to perform a target cognitive task versus change (improvement or decline).

For these various coding procedures, author LAR carried out the initial coding. Two additional authors (CMS and SAMS) subsequently identified items that they considered to be miscoded. Items in dispute were then reviewed and discussed until a consensus was reached, occasionally after discussing disputed items with authors DT, RNJ, and PKC.¹

Data Merged

Researchers from the different studies were asked to send item-level data—i.e., de-identified files that included responses to self-report items. For the ADNI, A4, and HABS-HD studies, we retrieved these data directly from the study databases. We included only the baseline (or first) assessment for each participant to ensure that observations were independent and to avoid the potential that previous exposure to a questionnaire affected subsequent responses. As we were interested in self-perceived cognitive functioning in neuropsychologically intact individuals, data from participants with prevalent or concurrently classified diagnoses of MCI, AD, or other dementia were excluded. Appendix A displays the definition of neuropsychologically intact or cognitively normal for each study.

(2). Item Overlap Examined: Determination of Identical, Equivalent, and Linking Items

After careful review of the database, a subset of authors (DT, LAR, SAMS, and RJ) noticed that some studies used multiple questionnaires (see Table 1), and there was overlap in questionnaires used among studies (see Table 2). These authors next sought to identify items from different questionnaires that were exactly the same, or identical. To be deemed identical, items were required to contain precisely the same item stems, response options, temporal referents (recall intervals), etc. Close examination of all items revealed that no identical items appeared on different questionnaires (i.e., the same exact item did not appear on more than one questionnaire). These authors then considered whether items with overlapping content were similar enough to be treated as equivalent in the analyses despite minor differences in wording. Such items contained overlapping content—i.e., addressed the same cognitive process, such as memory for recent events, but had slight differences in wording, and thus could not be deemed identical. Equivalent items were identified by consensus among authors DT, LAR, RNJ, SAMS. In some cases, response options for equivalent items were combined so that they would match across items. Appendix B presents sample equivalent items and recoding of response options. Items deemed equivalent were assigned a single code and subsequently treated as identical in study analyses. Both identical and equivalent items served as linking items in the analyses (i.e., were entered into the analysis as the same item). Our approach to linking test scores is known as the Non-Equivalent Anchor Test approach (NEAT; Dorans, Pommerich, & Holland, 2007), and is commonly used when different tests are given to non-equivalent groups (but there are common items administered between the groups). As noted above, some of our links involved complete tests, others involved a small number of items, and sometimes the linking items were generated ad hoc using the process for determining item equivalence.

Table 1.

Key Features of Participating Studies

Study name (Abbreviation)		Institutional affiliation(s)	Country (Language)	Research environment	Number of participants: neuropsychologically intact / total	Number of questionna ires	Final usable number of items
1.	Alzheimer’s Disease Center Clinical Core and Center for Brain Health (ADC-NYU)	New York University School of Medicine	United States (English)	Volunteer	245 / 788	1	4
2.	Alzheimer’s Disease Neuroimaging Initiative (ADNI)	Over 100 international affiliates: http://adni.loni.usc.edu	United States Canada Australia Germany United Kingdom (English; German)	Volunteer and memory clinic	938 / 1,729	2	60
3.	Amsterdam Dementia Cohort (Ams Dem Cohort)	VU University Medical Center	Netherlands (Dutch)	Memory clinic	569 / 569	3	27
4.	Anti-Amyloid Treatment in Asymptomatic Alzheimer’s Study (A4)	Multiple sites across the United States, Canada, and Japan: https://a4study.org/locations/	United States Canada Japan (English; Spanish; Japanese)	Community-based	5,667 / 6,945	3	18
5.	Australian Imaging Biomarkers and Lifestyle Flagship Study of Ageing (AIBL)	University of Melbourne	Australia (English)	Volunteer and physician referred	595 / 851	2	6
6.	Barcelona Group (Barcelona)	IDIBAPS, Hospital Clinic, Barcelona	Spain (Spanish)	Volunteer and memory clinic	186 / 186	1	26
7.	Bonn Memory Clinic (Bonn)	University of Bonn and the German Center for Neurodegenerative Diseases, Bonn	Germany (German)	Memory clinic	121/ 121	1	32
8.	Canadian Longitudinal Study on Aging (CLSA)	Multiple sites across Canada: https://www.clsa-elcv.ca	Canada (English; French)	Population-based	27,229 / 27,760	2	21
9.	Dartmouth Memory and Aging Study/Indiana Memory and Aging Study (Dart-Indiana)	Dartmouth Medical School, Indiana University School of Medicine	United States (English)	Volunteer and physician referral	141 / 271	8	173
10.	Einstein Aging Study (EAS)	Albert Einstein College of Medicine	United States (English)	Community-based	1,399 / 1,953	3	17
	EAS Substudy 1a				224 / 258	7	121
	EAS Substudy 2				292 / 447	1	41
11.	German Study on Ageing, Cognition, and Dementia in Primary Care Patients (AgeCoDe)	University of Bonn and 5 other German universities	Germany (German)	General practice registry-based	2,422 / 3,327	2	5
12.	Health and Aging Brain Study, Health Disparities (HABS-HD)	University of North Texas Health Science Center	United States (English)	Community-based	1,842 / 2,370	2	18
13.	Harvard Aging Brain Study (HABS)	Harvard Medical School, Brigham and Women’s Hospital and Massachusetts General Hospital	United States (English)	Volunteer	237 / 237	3	93
14.	Hellenic Longitudinal Investigation of Aging and Diet (HELIAD)	Aiginition Hospital, National and Kapodistrian University of Athens Medical School	Greece (Greek)	Population-based	1,550 / 1,943	3	15
15.	IMAP Caen Group	Inserm, University and Hospital of Caen	France (French)	Memory clinic	123 / 209	1	32
16.	Indiana Alzheimer’s Disease Center Cohort (IADC)	Indiana Alzheimer Disease Center; Indiana University School of Medicine	United States (English)	Memory clinic	226 / 350	2	60
17.	Leipzig Longitudinal Study of the Aged (LEILA 75+)	University of Leipzig	Germany (German)	Population-based	732 / 1,692	1	1
18.	Mayo Clinic Study of Aging (MCSA)	MCSA	United States (English)	Community-based	4,200 / 4,200	2	45
19.	Memory Clinic -Fundació ACE (Fundació)	Fundació ACE Barcelona Alzheimer Treatment and Research Center	Spain (Spanish)	Community-based	132 / 201	1	28
20.	Sydney Memory and Ageing Study (Sydney)	University of New South Wales	Australia (English)	Community-based	504 / 873	3	32
21.	University of Pittsburgh Study/ Monongahela-	University of Pittsburgh	United States (English)	Clinic-community volunteer	1,916 / 1,916	1	17
	Youghiogheny Health Aging Team (MYHAT)			sample and population- based	101 / 174		81
	Pittsburgh (Pitt) Substudyb					3
22.	Vanderbilt Memory and Alzheimer’s Center (VMAC)	Vanderbilt University School of Medicine	United States (English)	Memory clinic, physician referral, and community- based	115 / 191	1	51
23.	Victoria Subjective Cognitive Decline Study (Victoria SCDS)	University of Victoria	Canada (English)	Volunteer	42 / 42	4	53
24.	Wisconsin Registry for Alzheimer’s Prevention (WRAP)	University of Wisconsin- Madison	United States (English)	Community-based	1,282 / 1,538	2	35
TOTAL (including substudies)					53,030 / 61,141	65	1,112

^aA subset of participants (n = 260) received all seven questionnaires (ADL Abbrev, BRIEF-A, CAPM-C, CERAD-self, Einstein HSA, GDS Short, MIA); the remaining participants received three questionnaires (Einstein HSA, CERAD-self, GDS Short).

^bA subset of participants from the University of Pittsburgh (Pitt Substudy, n = 174) received all three questionnaires (MFQ, CFQ, SCCS); the remaining participants (from the MYHAT study) received one questionnaire (SCCS).

Table 2.

Key Features of Questionnaires Used by Participating Studies

Questionnaire/Item abbreviation		Questionnaire/Item full name	Study abbreviation(s)	References	Total number of original items	Number (percentage) of items relevant for analysisa	Final number of items retained
1.	ADL	Activities of Daily Living Rating Scale Self-Version	Dart-Indiana	Saykin, 1992	63	33 (52%)	33 (52%)
2.	ADL Abbrev	Activities of Daily Living Rating Scale Self-Version, Abbreviated	EAS Substudy 1	Saykin, unpublished	50	45 (90%)	43 (86%)
3.	AgeCoDe MQ	AgeCoDe Study Memory Questions	AgeCoDe	Jessen et al., 2007	2	2 (100%)	1 (50%)
4.	AIBL Screen	Melbourne Australian Imaging Biomarkers and Lifestyle Flagship Study of Ageing, Memory Screen Item	AIBL	Ellis et al., 2009	1	1 (100%)	1 (100%)
5.	Ams Dem Cohort CC	Amsterdam Dementia Cohort, subjective cognitive concerns screener	Ams Dem Cohort	van der Flier & Scheltens, 2018	5	5 (100%)	5 (100%)
6.	Blessed FAQ	Blessed Functional Activities Scale	HELIAD	Blessed et al., 1968	8	6 (75%)	6 (75%)
7.	BRIEF-A	Behavioral Rating Inventory of Executive Function- Adult Version	Dart-Indiana; EAS Substudy 1	Roth et al., 2005	75	36 (48%)	32 (43%)
8.	CAPM-C	Comprehensive Assessment of Prospective Memory- Section C	EAS Substudy 1	Chau et al., 2007	15	13 (87%)	10 (67%)
9.	CCI	Cognitive Change Index	ADNI; Ams Dem Cohort; Dart- Indiana; IADC	Rattanabannakit et al., 2016; Saykin et al., 2006	20	20 (100%)	20 (100%)
10.	CCI-40	Cognitive Change Index - extended 40 item version	EAS Substudy 2	Nester et al., 2022 [forthcoming]	41	41 (100%)	41 (100%)
11.	CDS-Q	Cognitive Difficulties Scale	IMAP Caen Group	McNair & Kahn, 1983	39	33 (85%)	32 (82%)
12.	CERAD-self	Consortium to Establish a Registry for Alzheimer’s Disease- Self version	EAS/EAS Substudy 1	Morris et al., 1989	36	22 (61%)	12 (33%)
13.	CFI	Cognitive Function Instrument, Self Report	A4	Amariglio et al., 2015a	15	12 (80%)	12 (80%)
14.	CFQ	Cognitive Failures Questionnaire	Pitt Substudy	Broadbent et al., 1982	25	17 (68%)	11 (44%)
15.	CLSA SCD/worry	Canadian Longitudinal Study in Aging, SCD/worry item	CLSA	Raina et al., 2009; 2019	1	1 (100%)	1 (100%)
16.	ECog Self	Everyday Cognition- Subject/Self-Report	ADNI; Dart- Indiana; HABS; IADC; MCSA	Farias et al., 2008	40	40 (100%)	40 (100%)b
17.	Einstein HSA	Albert Einstein Health SelfAssessment	EAS/EAS Substudy 1	Derby et al., 2013	20	6 (30%)	4 (20%)
18.	GDS Long	Geriatric Depression Scale - long version	Dart-Indiana; Victoria SCDS; HABS-HD	Yesavage et al., 1982	30	4 (13%)	4 (13%)
19.	GDS Short	Geriatric Depression Scale - short version	A4; EAS/EAS Substudy 1; HELIAD	Sheikh & Yesavage, 2005	15	1 (7%)	1 (7%)
20.	HELIAD SCI	HELIAD Subjective Cognitive Items	HELIAD	Margioti et al., 2020	7	7 (100%)	7 (100%)
21.	IQCODE Short	Short Form of the Informant Questionnaire on Cognitive Decline in the Elderly, self-report version	Dart-Indiana; Sydney	Jorm, 1994	16	15 (94%)	15 (94%)
22.	LEILA 75+ Questions	Leipzig Longitudinal Study of the Aged Questionnaire	LEILA 75+	Luck et al., 2010	4	4 (100%)	1 (25%)
23.	MAC-Q (a) d	Memory Complaint Questionnaire	A4; AIBL; ADC- NYU; Victoria SCDS	Riedel-Heller et al., 1999	6	6 (100%)	5 (83%)
24.	MAC-Q (b)	Memory Complaint Questionnaire	Sydney	Crook et al., 1992	6	6 (100%)	5 (183%)
25.	MATS	Memory and Aging Telephone Screen	Dart-Indiana; Victoria SCDS	Rabin et al., 2007	13	13 (100%)	12 (92%)
26.	MCSA Blessed	Mayo Clinic Study of Aging, adapted Blessed Memory scale	MCSA	Blessed et al., 1968; van Harten et al., 2018	6	5 (83%)	5 (83%)
27.	MFE-30	Memory Failures Everyday	Fundacio	Lozoya-Delgado et al., 2012	30	29 (97%)	26 (87%)
28.	MFQ	Memory Functioning Questionnaire	HABS; Pitt Substudy; WRAP	Gilewski et al., 1990	63, 64, or 54d	49 (78%); 50 (78%); or 39 (72%)	48 (78%); 49 (78%); or 35 (90%)
29.	MIA	Metamemory in Adulthood Questionnaire	EAS Substudy 1; Victoria SCDS	Dixon et al., 1988	108	35 (32%)	32 (30%)
30.	MMQ	Multifactorial Memory Questionnaire	Bonn Memory Clinic; CLSA	Troyer & Rich, 2002	57; 20e	32 (56%)	32 (56%)
31.	SCCS	Subjective Cognitive Complaint Scale-also called the Subjective Memory Scale	MYHAT; Pitt Substudy	Ganguli et al., 2010	30	27 (90%)	21 (70%)f
32.	SCD-Q: Part 1 MyCog	Subjective Cognitive Decline Questionnaire: Part 1 My Cognition	Barcelona Group	Rami et al., 2014	27	26 (96%)	26 (96%)
33.	SMCQ	Subjective Memory Complaints Questionnaire	HABS-HD	Youn et al., 2009; O’Bryant et al., 2022	14	14 (100%)	14 (100%)
34.	Amsterdam SCF	Subjective Cognitive Functioning	Ams Dem Cohort	Aalten et al., 2014	4	2 (50%)	1 (25%)
35.	SMDS	Subjective Memory Decline Scale	AgeCoDe	Jorm et al., 2001	4	4 (100%)	4 (100%)
36.	Squire	Squire Memory Self-Rating Questionnaire	Dart-Indiana	Squire et al., 1979	18	18 (100%)	18 (100%)
37.	STDA	Adapted from Structured Telephone Dementia Assessment	HABS	Go et al., 1997	7	7 (100%)	6 (86%)
38.	Sydney- SCQ Wave 1	Sydney Subjective Complaint Questions, Wave 1	Sydney	Slavin et al., 2010	18	16 (89%)	12 (66%)
39.	VMAC CCQ	Vanderbilt Memory and Alzheimer’s Center Cognitive Complaint Questionnaire	VMAC	Gifford et al., 2014	57	56 (98%)	51 (90%)
40.	WRAP Memory Item	Wisconsin Registry for Alzheimer’s Prevention, Memory Problem Item	WRAP	Nicholas et al., 2017	1	1 (100%)	1 (100%)
	TOTAL				963 g	690 h	635 i

^aItems relevant for the current analysis relate to the self-perceived cognitive functioning (see Methods and Results for details about eliminated items).

^bIn the HABS study, data were available for 39 of 40 ECog Self items.

^cThe MAC-Q was separated into versions a and b because of different timeframes referenced (high school or college for A4, AIBL, NYU, and Victoria SCDS studies versus five years ago for the Sydney study). Also, the ADC-NYU version of the MAC-Q contains a different item stem for item 4, which was not used in the current study (bringing the total number of MAC-Q items to one less for the NYU study).

^dThe Pitt Substudy, HABS, and WRAP study versions of the MFQ contain 64, 63, and 54 items, respectively.

^eThe MMQ has three subscales, of which the CLSA only includes one (i.e., Memory mistakes scale with 20 items).

^fIn the University of Pittsburgh MYHAT study, data were available for 17 of the 21 SCCS items.

^gThis total exclude 35 items from the CCI and GDS Short, as items from these questionnaires also appear on the lengthier versions of these questionnaires.

^hThis total excludes 21 items from the CCI and GDS Short, as items from these questionnaires also appear on the lengthier versions of these questionnaires.

ⁱThis total includes the 49-item version of the MFQ.

(3). Data Recoded

Items were recoded, as needed, so that for every item a higher response option corresponded to better self-perceived cognitive functioning. As an example, the original response options for item 56 from the Metamemory in Adulthood Questionnaire (MIA; Dixon et al., 1988), “My memory for dates has greatly declined in the last 10 years” ranged from 1= strongly disagree, 2 = disagree, 3 = undecided, 4 = agree, 5 = strongly agree. These were recoded—essentially reversed—so that the higher option categories matched better self-perceived cognitive functioning 1 = strongly agree, 2 = agree, 3 = undecided, 4 = disagree, 5 = strongly disagree. Also, item response categories were combined if a category had fewer than five respondents. This was the case for items that had Likert-type response scales where few participants endorsed categories at the extreme end(s) of the scale. These sparse extreme response categories were combined with the adjacent, less extreme category/categories, until the new combined category had five or more respondents.

(4). Assessed IRT Assumptions in the Reference Sample

The reference sample defines the latent trait in IRT analyses. The Mayo Clinic Study of Aging (MCSA), a large community-based sample of neuropsychologically intact older adults, served as the reference study for this analysis. The MCSA utilizes a fairly extensive assessment of self-perceived cognitive functioning that includes the Mayo Clinic Study of Aging, adapted Blessed Memory scale (Blessed et al., 1968; van Harten et al., 2018) and the Everyday Cognition- Subject/Self-Report (ECog Self; Farias et al., 2008), the latter of which assesses self-perceived cognitive functioning in multiple domains that include memory, language, visuospatial/perceptual ability, and executive functioning/divided attention. Moreover, the ECog Self is used in multiple studies, which facilitates linking of studies and items.

We used Mplus Version 8.5 (Muthén & Muthén, 2008–2017) for IRT analyses and the R statistical software package Version 4.0 (R Core Team, 2021) for data management and other study analyses. IRT is a latent variable technique used to describe the relationship between item responses and the latent trait presumed to underlie those responses, which in our case was the level of self-perceived cognitive functioning. The latent trait is assumed to be normally distributed with a mean of 0 and a standard deviation of 1 (Lord, 1980). By fitting IRT models to item data, self-perceived cognitive functioning performance can be estimated conditional on the observed item responses and their respective item parameters; additionally, IRT methods be used to generate scores that are comparable regardless of what items were utilized in a particular study.

Several key assumptions underlie the IRT framework, including unidimensionality of the measured trait, local independence, and monotonicity (McHorney & Cohen, 2000; Nguyen et al., 2014). We assessed the IRT assumptions in the MCSA sample by fitting a graded response IRT model (Samejima, 1977), which is typically used for ordered polytomous response data. All models were estimated using a Bayesian estimator (Haug, 2012; Winkler, 1993). A weakly informative prior distribution (N(0, 5)) was assumed for the model parameters. The Bayesian estimator allows for better performance in the small sample sizes that several studies provided (Hox et al., 2012; Lee & Song, 2004).

Unidimensionality Assumption

Based on the previous literature (Gifford et al., 2015; Snitz et al., 2011), we hypothesized that a unidimensional (or single factor) model might provide an adequate representation of the self-perceived cognitive functioning trait. In other words, the construct being measured, self-perceived cognitive functioning, is considered to have a single, dominant factor that accounts for item endorsement and covariation among the items (with small factors present beyond the major factor). This has been shown to be the case in previous work with cancer patients—where abilities and concerns about cognition loaded on a single factor and were conceptualized as a single dimension despite measuring different aspects of self-perceived cognitive functioning (Lai et al., 2014). We also considered the possibility that our data were multidimensional with strong factors beyond the major factor (i.e., differing cognitive domains might induce some meaningful secondary structure/residual covariance). For self-perceived cognitive functioning data, it is reasonable to expect that multidimensionality will arise from differences in content domain (e.g., some items assess the ability to remember recent conversations, others inquire about language problems, visuospatial ability, or language skills), temporal referents (e.g., “Compared to 10 years ago, do you notice change in…” versus “Do you currently have difficulty with ...”), or other key features of the questionnaire and its items. In a given study, any of these differences in item properties could contribute to subdomain structures and cause an item to deviate from strict unidimensionality. Such deviations from unidimensionality complicate the linking of the latent self-perceived cognitive functioning trait across multiple studies that use similar but nonequivalent questionnaires/items such that using a unidimensional-based calibration in the IRT model results in bias.

Prior to data analysis, we categorized items into cognitive subdomains and considered a bifactor model, which allows for additional covariation among a set of items beyond the general factor, based on these cognitive domains. Fitting a bifactor model proved to be an intractable problem in our analysis given the various possible subdomain structures (beyond cognitive domain) such as temporal referents or item content and structure. We therefore prioritized the unidimensional model, using two approaches to characterize the extent to which our inferences were likely to be biased due to unmeasured multidimensionality. Both approaches made use of data collected in the MCSA because, as noted, this was the study used to set the metric of our latent self-perceived cognitive functioning trait. Moreover, with one study, there was only one pattern of multidimensionality to be addressed for the item set.

Item-level Fit of Unidimensional and Bifactor Models: MCSA study.

The first method used to characterize threats to interpreting parameter estimates due to ignoring multidimensionality involved contrasting the posterior predictive p-values (PPP) for each of the items in the model under a unidimensional and a bi-factor model. Our bifactor model specification was based on the cognitive domain assignments. To assess unidimensionality in the MCSA, we fit unidimensional and bifactor item response models, and examined PPP for each item in the model. The PPP compares the χ² value from the model generated data to the χ² from the observed data and is approximated by the proportion of times the model generated data χ² is greater than the observed data χ² (Gelman, 2013; Meng, 1994). A low PPP implies that it is unlikely that the observed data derive from a data generating model that is represented by the model used in the analysis. Conventionally, PPP less than .05 indicate poor fit. The distribution of PPP for all items is shown in Appendix C, Figure 1. This figure reveals that, overall, the fit was good. The mean PPP was higher in the unidimensional case relative to the bifactor case (.38 vs .28, respectively). In both models for the MCSA, all items had PPP greater than .05. Therefore, we inferred that for the MCSA self-perceived cognitive functioning items, the pattern of correlations was consistent with a unidimensional model.

Differences in Common Latent Trait Estimates Under Unidimensional and Multidimensional Models.

The second method used to probe the level of misfit introduced with the assumption of unidimensionality involved examining the estimated factor scores derived from a unidimensional and a priori specified bifactor model. Results of these analyses are summarized in Appendix C, Figure 2 (left and right panels). These plots show the scatterplot of factor scores (left) and the difference in factor scores as a function of the average of the two factor scores (right). In our work, we typically consider absolute differences in factor scores of greater than 0.3 to be large and meaningful (Gibbons et al., 2017). Only 3% of the participants in the MCSA had a greater than 0.3 difference in factor scores. Therefore, the unidimensional model-derived factor score estimate would be deemed sufficiently precise for 97% of the sample. Further inspection of Appendix C, Figure 2 revealed that there was a region where the unidimensional factor scores seemed to produce a biased high estimate of the latent trait: values greater than +2 on the latent trait distribution, or the expected top 2.5% of scores. Thus, bias was apparent in the estimates from the unidimensional models, but it was restricted to the upper tail of self-perceived cognitive functioning (i.e., best performers/lowest cognitive concerns). As higher values on the latent trait indicated better self-perceived cognitive functioning, these 2.5% of respondents were of the least interest, clinically, if the goal was to identify individuals with higher levels of concern about their cognitive functioning.

Item-level Fit of Unidimensional and Bifactor Models: All Studies.

Further support for the assumption of unidimensionality came from examining the PPP for all items included in our analytic models, in all studies, from a unidimensional model and a bifactor model (Appendix C, Figure 3). The secondary structure was specified according to a priori assignment of items to specific domains based on manifest item content. For the PPP obtained, assuming unidimensionality, 96 of 1,112 had PPP less than .05, suggesting that for the vast majority of items, the data were consistent with the assumption of a data generating model that was unidimensional. For the bifactor models, 76 of 1,112 had PPP less than .05, suggesting that for this fraction of items even our bifactor specification was insufficient to account for the observed covariation. For these items, it is likely that additional covariation among items other than due to the manifest cognitive domain was responsible for the misfit, and we suspect that other item features (e.g., temporal referent, format and range of response options, whether items tapped ability or change) could be responsible, or that our a priori domain assignments did not conform to observed patterns of covariation among the items.

Local Independence and Monotonicity Assumptions

The next assumption, local independence, means that for a given latent trait (in our case self-perceived cognitive functioning), item responses are unrelated (Edwards et al., 2018); the assumption is usually tested for pairs of items (Kim et al., 2011) and meeting this assumption can be an additional indication of adequate model fit. In the MCSA study, we checked for and eliminated items with factor loadings > 1, which implies that there was a strong residual correlation or logical dependency with another item. The monotonicity assumption means that as the trait level increases, the probability of a correct response also increases (Kang et al., 2018; Nguyen et al., 2014)—i.e., the higher the factor score the more likely a respondent is to endorse concerns about cognitive functioning. In the MCSA sample, we checked for and eliminated items that demonstrated a common factor loading qualitatively different from theoretical expectations (loadings < 0), which implies that the directionality of the item is incorrect and may indicate the measurement of a distinct construct.

(5). Decision to Utilize a Unidimensional Model

In sum, across a set of models assuming unidimensionality and another set assuming multidimensionality (modeled with the bifactor approach), we compared item level fit and values estimated for latent traits. We found that item level fit was superior under the assumption of multidimensionality. However, we also found that in our reference sample, the item fits (PPP) were consistent with the assumption of unidimensionality, and in our overall sample only 9% of items demonstrated poor fit (PPP <. 05) under the assumption of unidimensionality, and 7% of items demonstrated poor fit in models intending to capture multidimensionality. While this result favors the bifactor approach, it also reveals that the bifactor approach alone is insufficient to account for the poor fit of all items in our large item set, and the gain in fit of a bifactor approach over a unidimensional approach was not large. We then examined differences in factor score estimates obtained in the reference sample under a unidimensional and bifactor model. For 97% of the sample, the differences were less than 0.3 standard errors, an acceptable level to assume equivalence. Bias was apparent in the estimates from the unidimensional models, but it was restricted to the upper tail of self-perceived cognitive functioning performance (i.e., best performers). Based on these results, we concluded that there was little evidence of important bias that would result from using a unidimensional model, and we proceeded with this approach.

(6). Examination of Differential Item Functioning

We performed a brief analysis of uniform differential item functioning (DIF) with respect to study to determine whether participants with the same latent trait level across different studies differed in their response patterns on identical items. To be included in the DIF analysis, an item needed to be present across at least two studies. For most studies, the number of linking items was small and not conducive to formal DIF testing. This limited us to studies that used the same questionnaire. The studies that were most useful for a DIF analysis were the five that included the ECog Self: MCSA, ADNI, Dartmouth Memory and Aging Study/Indiana Memory and Aging Study (Dart-Indiana), HABS, and the Indiana Alzheimer’s Disease Center Cohort (IADC).

(7). Saved MCSA Item Parameters to Item Bank

An item bank is a collection of item parameters calibrated by using IRT models. It is created by saving the estimated item parameters from the IRT model with new items added to the bank from calibrating subsequent studies. Item banks can be used by instrument developers to refine existing measures by selecting the “best” set of questions for a particular purpose or for developing new instruments including using computerized adaptive testing (CAT) platforms and/or static, fixed-length short forms (Lai et al., 2003; 2014). The two sets of parameters were the item discrimination (or measurement slope) and item difficulties or thresholds (i.e., locations on an item-level latent response variable at which the most likely response moves from a lower to a higher value). The item bank was initiated with parameter estimates, taken as the medians of the posterior distribution for each of the IRT item parameters, from the MCSA.

(8). Item Linking/Harmonization in MCSA

We accomplished the statistical harmonization of questionnaires using a sequenced estimation of IRT models. There are two broad approaches to item linking—common item anchoring and common group equating. In common group anchoring, an assumption is that the groups are equivalent on the trait of interest—and the different questionnaires can simply be linked. However, we could not make that assumption given inherent differences in our samples for key sociodemographic and other variables. Instead, we utilized the common items anchoring approach, where there are identical or equivalent items across different questionnaires, and we fit a series of IRT models using item banking. The linking transformation served to place the items on the same scale (Vale, 1986).

To perform statistical harmonization (or linking) of the questionnaires, we used co-calibration (i.e., putting the questionnaires on the same scale) based on IRT in a series of steps. We initiated the harmonization sequence with the MCSA and its two questionnaires, ECog Self and MCSA adapted Blessed Memory scale (MCSA Blessed; Blessed et al., 1968; van Harten et al., 2018). This approach anchored the self-perceived cognitive functioning latent trait to a community-based sample with no neuropsychological impairment. As the reference population, the mean trait level in MCSA was set to 0 (with SD set to 1), affording comparability with other studies as they were harmonized. As an example, if the mean trait level for a study entered after MCSA was 0.5, this would indicate that the average participant in that study was .5 SD higher on the self-perceived cognitive functioning trait than the average MCSA participant. Figure 2 shows the ordering of studies in the harmonization process, with MCSA and its questionnaires entered first. The order in which subsequent studies were harmonized after MCSA was determined by considering item overlap with previous studies and sample size. As described above, response categories for certain items were combined. Decisions related to combining response categories for a given item were made for the first study in the harmonization process to use the item. For example, for the ECog Self item “finding my way around a familiar neighborhood,” the response options of “consistently a little worse” and “consistently much worse” were combined. These decisions were then carried forward to subsequent studies that used the item (i.e., to the four other studies that used the ECog Self, see Table 2).

(9). Algorithm Carried Through to Subsequent Studies

After MCSA, we analyzed other studies sequentially. If subsequent studies included items already calibrated in the item bank, we constrained those item parameters to values in the item bank, and freely estimated parameters for items not already in the bank. New items and their parameters were thus added to the item bank. This approach (banking and linking) required that—apart from the reference study—each subsequent study had at least one item in common with a previously calibrated item in the bank. This implies that studies with no overlap of items could not be included in the linking process, and to improve the quality of the estimates we needed to balance the size of the study and number of items represented in the bank in choosing the order in which items were added. As was done in MCSA (discussed above), we performed data quality checks and preliminary model checks for items with measurement slopes that corresponded to standardized factor loadings that were problematic (i.e., < 0 or >1). This was an additional quality check on whether the direction and potential recoding for each item were done correctly. It also enabled us to identify items where we had made an error, necessitating reverse coding of items that we had erroneously failed to reverse code. In this way, we were able to use the signs for the loadings as a further check on the directionality of each item as included in the final model. Beyond coding discrepancies, several other items were excluded via two iterations of a cycle of excluding items, re-harmonizing remaining items, and examining factor loadings until all factor loadings were between 0 and 1 for each study.

(10). Calculated Item Information and Identified Items with the Highest Information Values (“Top” Items)

Item information was determined by the derivative of the item response function—i.e., the increasing probability of endorsing an item response in a higher category as a function of the underlying trait level. The amount of information at a given latent trait level is the inverse of its standard error of measurement (SEM) squared; therefore, the larger the amount of information provided by the item, the greater the precision of the measurement (Hays et al., 2000). The amount of information (and precision) varies across levels of the latent trait. The location on the latent trait with the maximum amount of information for a binary or dichotomous item is where the probability of positively or negatively endorsing a response category is equal.

We plotted item information curves (Hambelton & Cook, 1977), with item information plotted against the latent trait. Item information curves are useful because the item parameters and the latent trait are on the same scale, and items can be selected based on their measurement ability in regions of interest on the latent trait (Baker & Kim, 2017). For example, measuring self-perceived cognitive functioning on the “higher” end might be of interest when studying healthy older adults living in the community, whereas the “lower” end might be more interesting in individuals who present to a memory clinic with concerns about cognition. Some items or questionnaires may provide more information in one region of the latent trait and less in a different region. For the current analysis, we determined several regions of interest on the latent trait. The first region was from 0 to +1 SD above the mean of the latent trait, which may capture individuals with better self-perceived cognitive functioning—i.e., no or few concerns about cognitive function. A second region was from the mean to -1 SD below the mean, which may capture individuals with a mild level of self-perceived cognitive functioning concerns. A third region was from -1 to -2 SDs below the mean of the latent trait, which may capture individuals with worse self-perceived cognitive functioning—i.e., a moderate to high level of cognitive concerns. From the regions of interest, we identified “top items” with the highest information values.

Results

Participant and Study Characteristics

Data were available for 61,141 participants. Because we were interested in individuals with normal performance on neuropsychological tests, we excluded one of the studies that only contributed data for cognitively impaired individuals (i.e., Pre-Al, Hôpital de la Salpêtrière, AP-HP). We also excluded subsets of cognitively impaired participants from 18 of the 24 remaining studies (see Table 1). After removing these individuals (n = 8,111), the final sample contained 53,030 neuropsychologically intact participants with data on self-perceived cognitive functioning items. Table 1 presents key features of the 24 participating working group studies. Two of the participating sites (i.e., Einstein Aging Study and University of Pittsburgh) had substudies as shown in Table 1 and Figure 2, but for the purposes of this study were considered a single site/study, unless otherwise specified. Contributing studies had sites in the U.S., Canada, Australia, Germany, United Kingdom, the Netherlands, Spain, Greece, France, and Japan. In addition, various research environments were represented including memory clinics (n = 4), community-based samples (n = 7), volunteer samples (n = 3), population-based samples (n = 3), a general practice registry (n = 1), and mixed sampling approaches (n = 6).

Questionnaire Characteristics and Linking Items

Forty questionnaires qualified for inclusion (Tables 1 and 2). As shown in Table 1, for the 24 primary studies (i.e., excluding the Einstein Aging Study and University of Pittsburgh substudies), the number of questionnaires used per study ranged from 1 to 8 (mean 2.3; median 2.0). Also, there was considerable variability in the number of usable items per study, ranging from 1 to 173 (mean 36.2; median 27.5; inter-quartile interval 46.5–17.0). There was minimal overlap of questionnaires across studies, with the majority (n = 30, 75%) used only within a single study. Appendix D shows the number of studies in which a given questionnaire was utilized. In terms of item content, as shown in Table 3, over half of the items related to memory (57%), followed by executive function (16%), attention/working memory/processing speed (10%), and language (9%).

Table 3.

Percentage of 601 Items Within Each of Eight Cognitive Domains

Cognitive domain	Number of items (Percentage)	Description
memory	344 (57%)	includes short-term/long-term/episodic/semantic/prospective memory, and learning new information
executive function	97 (16%)	includes organizing, planning, switching, initiating, multi-tasking, reasoning, judgment, problem solving, decision-making, handling emergencies, impulsivity and self-regulation, clarity of mind, learning or operating machinery, handling money, and self-awareness of problems
attention working memory processing speed	63 (10%)	includes basic attention, sustained attention, focused attention, concentration, divided attention, and alertness
language	52 (9%)	includes expressive and receptive language, word finding, reading, and spelling
visuospatial skills	23 (4%)	includes visuoperception, route finding, and directional orientation
orientation	9 (2%)	includes orientation to person, time, place, or situation
calculation	7 (1%)	includes basic calculation and arithmetic tasks
general cognitive ability	6 (1%)	includes memory and other thinking abilities grouped together in a single item

Table 2 presents key features of the questionnaires including the total number of original items from each questionnaire with a total across questionnaires of 963 items (this total includes the 64-item version of the MFQ). In the penultimate column, we present the number of items relevant for the current analysis because they relate to self-perceived cognitive functioning (total n = 690). We excluded 273 items that related to other characteristics such as mood, personality, health complaints, basic activities of daily living, use of memory/other cognitive strategies, etc. Of the 690 items, 55 were subsequently discarded during the data analysis process, leaving 635 items. For example, we dropped items initially included but later determined to have critical content or response category limitations such as: “I am impulsive” and “Do you fail to listen to people’s names when you are meeting them?”. These items could not be categorized within the eight cognitive domains and could capture something other than self-perceived cognitive functioning. Similarly, items such as “If your memory was 100% when you were 30 years old, what % would you say it is now?” and “How long ago did your memory problems start?” had free-response formats not easily handled by our modeling approach. As noted above, during statistical harmonization, we also dropped items with loadings < 0 or > 1.

As described above, we also considered items with overlapping content that were similar enough to be treated as equivalent in the analyses despite minor differences in wording. Equivalent items were given the same item number and treated as a single item in the analysis; therefore, we lost one item for every replicate of an equivalent item. There were 27 sets of equivalent items: 22 items had one replicate, 3 items had two replicates, and 2 items had three replicates. In total, 34 (22 + 6 + 6) replicates were lost from the item total because they were considered equivalent items. Thus, we began with 635 items but after the equivalence process that yielded 11 sets of equivalent items, our final item count was 635 – 34 = 601. These equivalent items (n = 27) in combination with the identical items (n = 220), comprised the linking items (n = 247), which were used to link studies in the IRT analyses. As noted above, identical items were items from questionnaires that were used in more than one study.

Every study had at least one item in common with another study (equivalent or identical)—i.e., all studies had at least one linking item. Appendix E shows the number of linking items between pairs of studies. As shown in bold in Appendix E, the highest number of linking items between two studies was 60 for ADNI and the IADC, which both utilized the ECog Self and the Cognitive Change Index (CCI). For study pairs that had linking items among them (e.g., MCSA and IADC had 40, Memory Clinic - Fundació ACE and the Bonn Memory Clinic had one), the mean number of linking items between studies was 11.1 (SD = 12.6). As shown in Appendix F, each study shared linking items with an average of 6.2 other studies SD = 3.2 (range 1 to 14).

Figure 2 shows the ordering of studies for the IRT analysis and overlap among self-perceived cognitive functioning items across studies including the substudies. In the top panel, the horizontal black and white bars consist of multiple dots that represent items administered within a study. The bars that align vertically represent identical items that come from the same questionnaire. The bars that are black, at the top of the vertical-colored regions, indicate the study used to calibrate the items for the item bank. The white bars are identical to items in the black bars with which they vertically align, and represent items already calibrated by the top-most study above it. Equivalent items are single dots found within a colored region that indicate items from different instruments merged for study analysis. Items that are within a colored region (both black dots and white bars) provide the linkage for a study between previously calibrated items and items to be calibrated.

As shown in the top panel of Figure 2, there was very little item overlap across studies, and no single item represented across all studies. In some cases, a small number of items provided the linkage between specific studies. For example, the Barcelona Group had two items that linked to other studies. This is represented by a circle around two dots: (1) the Barcelona Group study was linked to the Bonn Memory Clinic and Canadian Longitudinal Study on Aging (CLSA) by a single item (single dot in the tan region); and (2) the Barcelona Group was linked to the Vanderbilt Memory and Alzheimer’s Center (VMAC) study by a single item (single dot in the purple region). Another example is the three items from the IMAP Caen Group, which were equivalent to those from the Bonn Memory Clinic (three dots circled in the tan region). A final example from the figure is the VMAC study, which had links to 14 other studies. Here we highlight the single dot circled in the teal region, which is an item from the VMAC study that is equivalent to an item from the Einstein Aging Study.

The bottom panel displays additional detail for the portion of the top panel enclosed in a rectangle. In this panel, each dot reflects an item from a given questionnaire (dots are ordered so that all of a given questionnaire’s items are grouped together). MCSA was the first study to be harmonized and is at the top (MCSA has the ECog Self and MCSA Blessed). ADNI was then harmonized and has the ECog Self and CCI questionnaires. The ECog Self provides the link to MCSA. This is the first encounter with the CCI, and it will therefore be calibrated with the ADNI sample. The IADC has two questionnaires (ECog Self and CCI), both of which were previously calibrated; therefore, the IADC does not provide any new questionnaires to be calibrated. The HABS will be linked through the ECog Self and will calibrate the Memory Functioning Questionnaire (MFQ) and Adapted from Structured Telephone Dementia Assessment (STDA).

Differential Item Functioning Analysis

As discussed above, to lend credibility to the harmonization process, we conducted a DIF analysis with respect to study on the subset of items identified as linking items from the ECog Self in five studies (n = 5,742). As shown in Appendix G, we found evidence of DIF using a statistical significance criterion for flagging items with DIF; 36 of 160 study-item pairs (23%) had evidence of DIF, which is not surprising given the large sample size. Accounting for DIF in the estimation of latent traits accounted for between 2% and 19% of the DIF-naïve group differences in the self-perceived cognitive functioning trait. Moreover, the absolute differences in the mean level of the ECog Self common latent trait were between 0.01 and 0.03 standard deviation units (Appendix G), which are considered very small (i.e., the latent trait estimates do not meaningfully change after accounting for DIF).

Item Content Examination

Higher scores on the underlying latent self-perceived cognitive functioning trait represented better self-perceived cognitive functioning. In support of the idea that these items actually measured the trait of interest, the vast majority of items (575 of 601) had standardized factor loadings greater than 0.3, and the median (and inter-quartile interval) was 0.61 (0.50 – 0.71). The ranges over which we present item information values for selected items reflected different levels of the latent trait, and these tables (see below) highlight items that might be most relevant for individuals within a given range.

Tables 4a and 4b present the 30 items with the highest information values and lowest SEM in the underlying latent trait ranges of −2 to −1 SDs and 0 to +1 SD, reflecting psychometrically sound items likely to be endorsed by participants with high and low levels of concern about their cognitive functioning, respectively. Items in the middle range (−1 to 0 SD) almost complete content overlap with items in the −2 to −1 or 0 to +1 SD ranges, and are presented in Appendix H. Of the top 15 items with good measurement precision in the higher concern range of −2 to −1, memory accounted for 80% of items (n = 12). By contrast, of the top 15 items in the higher concern range of 0 to +1, there was more variability in cognitive domains represented, with 33% (n = 5) of items related to memory, and the remaining items related executive function (n = 5), language (n = 3), calculation (n = 1), and attention and concentration (n = 1). Across the ranges of interest, most items (70%, n = 21) assessed perceptions of change (“C”) in ability or function from some prior state or level, while 30% (n = 9) assessed current ability (designated “A”). With respect to response option types, of the top 30 items 67% (n = 20) used Likert-type and rankordered/categorical item scales; notably, all of the dichotomous items (n = 10) came from the higher concern range of −2 to −1.

Table 4a.

15 Items with the Smallest SEM in the Range of -2 to -1

	Questionnaire abbreviation	Cognitive domain	Item stem	Response optionsa	Temporal referent	Ability versus change	SEM; Information
1.	VMAC CCQ	memory	Do you think you have problems with your memory?	no; yes	current	A	0.37; 7.20
2.	VMAC CCQ	memory	Do you have problems with your memory compared to the way it was 5 years ago?	no problems; some minor problems; major problems	compared to 5 years ago	C	0.39; 6.53
3.	VMAC CCQ	memory	Do you feel that your everyday life is difficult now due to your memory decline?	no; yes	current	A	0.42; 5.65
4.	VMAC CCQ	memory	Do you have difficulty remembering a conversation from a few days ago?	no; yes;	current	A	0.42; 5.56
5.	ADL Abbrev	memory	Remembering what I intended to do	much better; slightly better; no change;	ability compared to 5 years ago	Cb	0.43; 5.29
				slightly worse;
				slightly to moderately worse; moderately worse; much worseb
6.	CFI	executive	Compared to one year ago, do you have more difficulty managing money (e.g., paying bills, calculating change, completing tax forms)?	no; maybe; yes;	compared to one year ago	C	0.44; 5.27
7.	CFI	memory	Compared to one year ago, do you feel that your memory has declined substantially?	no; maybe; yes;	compared to one year ago	C	0.44; 5.09
8.	HELIAD SCI	memory	Do you have difficulty remembering things that you just read or heard?	no problems; problems	current	A	0.46; 4.69
9.	CFI	attention	Do you have more trouble following the news or the plots of books, movies, or TV shows?	no; maybe; yes;	with reference to one year ago	C	0.47; 4.46
10.	SMCQ	memory	Do you have difficulty in remembering an appointment made a few days ago?	no; yes	current	A	0.48; 4.41
11.	VMAC CCQ	memory	Do memory problems make it harder to complete tasks that used to be easy?	no; yes	current	A	0.48; 4.39
12.	VMAC CCQ	memory	Do you consider your own memory to be worse than others that are your same age?	no; yes	current	A	0.48; 4.26
13.	VMAC CCQ	memory	Do you have complaints about your memory in the last 2 years?	no; yes	in the last 2 years	A	0.49; 4.17
14.	VMAC CCQ	memory	Do you lose objects more often than you did previously?	no; yes	current	C	0.49; 4.13
15.	CFI	visuospatial	Are you more likely to become disoriented, or get lost, for example when traveling to another city?	no; yes	one year ago	C	0.50; 3.94

^aHighlighted response options were combined for purposes of data harmonization due to the categories having fewer than five respondents.

^bThe ADL Abbrev scale uses a time referent of compared to 5 years ago to capture current ability level on certain tasks, skills, or problem areas in relation to previous ability. As such, even though labeled as “C”, indicating that it assesses change, the ADL Abbrev also taps into ability.

Note. SEM = Standard Error of Measurement; ADL Abbreviated = Activities of Daily Living Rating Scale Self-Version, Abbreviated; CFI = Cognitive Function Index; HELIAD SCI = HELIAD Subjective Cognitive Items; SMCQ = Subjective Memory Complaints Questionnaire; VMAC CCQ = Vanderbilt Memory and Alzheimer’s Center Cognitive Complaint Questionnaire

Table 4b.

15 Items with the Smallest SEM in the Range of 0 to +1

	Questionnaire abbreviation	Cognitive domain	Item stem	Response optionsa	Temporal referent	Ability versus change	SEM; Information
1.	ADL Abbrev	executive	Focusing on goals or carrying out a plan	much better; slightly better;b	ability compared to 5 years ago	Cb	0.50; 4.08
				no change;
				slightly worse slightly to moderately worse; moderately worse; much worse
2.	CFI	language	Do you have more trouble recalling names, finding the right word, or completing sentences?	no; maybe; yes;	reference to one year ago	C	0.52; 3.77
3.	ADL Abbrev	memory	Remembering appointments or meetings	much better; slightly better;	ability compared to 5 years ago	C	0.54; 3.48
				no change;
				slightly worse slightly to moderately worse; moderately worse; much worse
4.	MMQ	memory	I worry about my memory ability	strongly disagree; disagree; undecided; agree; strongly agree	feelings over the past two weeks	A	0.55; 3.29
5.	ADL Abbrev	attention	Remaining on task and not getting distracted by external stimuli	much better; slightly better;	ability compared to 5 years ago	C	0.57; 3.03
				no change;
				slightly worse slightly to moderately worse; moderately worse; much worse	ability compared to 5 years ago	C	0.57; 3.03
6.	ADL Abbrev	executive	Shifting easily from one activity to the next	much better; slightly better;	ability compared to 5 years ago	C	0.60; 2.75
				no change; slightly worse;
				slightly to moderately worse; moderately worse; much worse
7.	ADL Abbrev	executive	Making decisions on everyday matters	much better; slightly better;	ability compared to 5 years ago	C	0.61; 2.65
				no change;
				slightly worse; slightly to moderately worse; moderately worse; much worse
8.	MAC-Q (a)	memory	Remembering specific facts from a newspaper or magazine article you have just finished reading	much better now; somewhat better now;	as compared to high school or college	C	0.62; 2.63
				about the same; somewhat poorer no much poorer now
9.	MAC-Q (a)	memory	Remembering the name of a person just introduced to you	much better now; somewhat better now; about the same;	as compared to high school or college	C	0.62; 2.59
				somewhat poorer now; much poorer now
10.	CFI	memory	Compared to one year ago, do you find that you are relying more on written reminders (e.g., shopping lists, calendars)?	no; maybe; yes;	reference to one year ago	C	0.62; 2.59
11.	ADL Abbrev	executive	Reasoning through a complicated problem	much better; slightly better; no change; slightly worse;	ability compared to 5 years ago	C	0.63; 2.56
				slightly to moderately worse; moderately worse; much worse
12.	ADL Abbrev	language	Expressing myself through speech	much better; slightly better;	ability compared to 5 years ago	C	0.63; 2.54
				no change; slightly worse slightly to moderately worse;
				moderately worse; much worse
13.	ADL Abbrev	executive	Organizing and managing my medication schedule	much better; slightly better; no change;	ability compared to 5 years ago	C	0.63; 2.54
				slightly worse; slightly to moderately worse; moderately worse; much worse
14.	ADL Abbrev	language	Understanding instructions or directions	much better; slightly better;	ability compared to 5 years ago	C	0.63; 2.54
				no change;
				slightly worse; slightly to moderately worse; moderately worse; much worse
15.	ADL Abbrev	calculation	Handling everyday arithmetic problems (knowing how much food to buy, how long it’s been between visits from family and friends)	much better; slightly better;	ability compared to 5 years ago	C	0.63; 2.49
				no change; slightly worse; slightly to moderately worse; moderately worse; much worse
				slightly worse; slightly to moderately worse; moderately worse; much worse

^aHighlighted response options were combined for purposes of data harmonization due to the categories having fewer than five respondents.

^bThe ADL Abbrev scale uses a time referent of compared to 5 years ago to capture current ability level on certain tasks, skills, or problem areas in relation to previous ability. As such, even though labeled as “C”, indicating that it assesses change, the ADL Abbrev also taps into ability.

Note. SEM = Standard Error of Measurement; ADL Abbreviated = Activities of Daily Living Rating Scale Self-Version, Abbreviated; CFI = Cognitive Function Index; MAC-Q (a) = Memory Complaint Questionnaire; MMQ = Multifactorial Memory Questionnaire.

Figure 3 presents the item level information plots for all 601 items (i.e., those with both high and low information values). The items that appear in blue have the highest information in the two ranges, and these are the items presented in Tables 4a and 4b). Also, items that are dichotomous have only one threshold and therefore a single peak. Polytomous/Likert-scaled items have multiple thresholds and therefore can have multiple peaks. For polytomous items with multimodal curves, information typically covers a broader range—i.e., there is a higher level of information over a broader range of ability level. Notably, most of the peaks fell within the 0 to −2 range, reflecting worse self-perceived cognitive functioning, consistent with the design of these items in terms of capturing self-perceived difficulties and/or declines in cognitive functioning.

Appendix I presents the 15 items with the highest SEM in the underlying latent trait range of −4 to +4 SDs, reflecting items with weak psychometric properties across the range of the latent trait. For the items included in this analysis, additional parameters and features (e.g., item discrimination, item difficulty thresholds, summary of item content) are available upon request from the corresponding author.

Discussion

Overview

The current study harmonized self-perceived cognitive functioning questionnaires across international aging studies, with over 53,000 neuropsychologically intact participants, using an IRT item banking approach. The measurement properties of 601 items from studies of various sizes and settings were characterized, and items with the strongest psychometric properties were identified. The challenges we encountered in linking items and harmonizing these data speak to the effort involved in reviewing item content, determining equivalences among items, managing the data files, and performing statistical analyses across the 24 studies and 40 questionnaires. Results may lead to the development of new questionnaires guided by psychometric properties, content, and other important features of items in our item bank.

Dimensionality

We were able to replicate and expand upon earlier work in linking self-perceived cognitive functioning items and placing them on a common scale using IRT (Gifford et al., 2015; Snitz et al., 2011). We selected a unidimensional measurement model as our final modeling strategy. Overall, results did not provide convincing evidence to go forward with a bifactor model over a unidimensional model based on the estimated factor scores (which did not significantly differ between the two models) and the PPP for the individual items (which suggested a good fit for both models). As the secondary structure did not markedly impact scores, loadings, or fit, we determined that the single factor model was appropriate for modeling self-perceived cognitive functioning. However, we cannot rule out that a different approach with a multidimensional factor structure for the multiple domains of self-perceived cognitive functioning might have presented a more nuanced representation. Nonetheless, we found the specification and estimation of multidimensional models intractable given the sparse coverage of domains and limited overlap of items across studies. As we evaluated the approach in this study, we found that if the structure of the secondary or specific factors was different in different studies, the meaning (and expected population values) of secondary factor loadings were also different. The implication is that for each study, the common and specific factors must be included in the linking operation, and this is an important area for future methodological and applied research.

The finding that a unidimensional model fit the data lends support to the idea that self-perceived cognitive functioning can reasonably be conceptualized as a unitary construct for measurement purposes, regardless of the specific domains tapped by items on a given questionnaire. In fact, in the original criteria set forth by Jessen and colleagues (2014), the authors stated that “cognitive” refers to any cognitive domain and is not restricted to memory. The term cognitive, as opposed to memory, was specifically chosen for the SCD criteria because the first symptoms of AD may not be limited to memory decline, and older adults often report memory decline when they actually experience decline in other cognitive domains such as language or executive functioning and vice versa (e.g., reporting poor memory when the difficulty is actually in language or attention). Additionally, work by van Harten and colleagues (2018) showed that all ECog Self subdomains and multidomain scores (generated from items across cognitive domains) were associated with MCI. In this study, the multidomain score was slightly better than the separate domain scores, lending support to the idea that a general self-perceived cognitive functioning score rather than a specific domain score might be most predictive of underlying neurodegenerative change (van Harten at al., 2018).

Features of Psychometrically Most Robust Items

Examination of the most robust items—those with the lowest SEM/highest information values from the specified ranges of interest—revealed several notable features. Top items in the low self-perceived cognitive functioning (high concern) range almost exclusively related to memory, both general (e.g., memory problems or decline relative to one year or five years ago) and specific (e.g., difficulty remembering conversations, remembering what you intended to, remembering appointments). By contrast, top items in the high self-perceived cognitive functioning (low concern) range assessed various aspects of cognition in addition to memory, most notably executive functioning (e.g., focusing on goals, decision making, shifting between activities, managing a medication schedule) and language (e.g., word finding, self-expression, understanding instructions). Additionally, several items in the low self-perceived cognitive functioning (high concern) range tapped into the consequences of the memory problems (e.g., “Do you feel that your everyday life is difficult now due to your memory decline?”; “Do memory problems make it harder to complete tasks that used to be easy?”), which was not the case for the high self-perceived cognitive functioning (low concern) range. Together these results raise the possibility that different types of items might be most relevant for neuropsychologically intact older adults in different stages of SCD. If this is the case, then ultimately it may be possible to select items based on their ability to distinguish between (or capture concerns most associated with) specific levels of SCD. For example, items from the higher concern range, which focused on memory and included various comparisons and consequences or impact on everyday life, might be most suitable for identifying those at risk for future dementia. By contrast, items from the lower concern range, which related mainly to higher order executive cognitive abilities (e.g., carrying out plans, making decisions, shifting set, being distracted, managing medications) or language (e.g., wording finding, self-expression), might be more suitable for older adults at lower risk of progression to AD. These various possibilities will need to be investigated through prospective data collection and longitudinal follow-up of neuropsychologically intact older adults.

Another finding was that items from the low self-perceived cognitive functioning (high concern) range tended to have fewer response options. Almost all items (14 of 15) were either dichotomous or had three response options (e.g., no; maybe; yes), with the reverse pattern occurring for the high self-perceived cognitive functioning (low concern) range where almost all items (13 of 15) utilized 5- and 7-point Likert-type scales. Follow-up research should determine whether fewer response options accurately capture the experiences of neuropsychologically intact older adults with worse self-perceived cognitive functioning (such as those presenting to a memory clinic). By contrast, a wider range of options may be necessary to adequately assess concerns among those with better self-perceived cognitive functioning, as it may allow these individuals to endorse subtle levels of difficulty or change across time. Given that self-perceived cognitive functioning likely falls along a continuum, Likert-type scales with a sufficient number of response options (i.e., 5 to 7) might be most appropriate to capture variability and change over a broad range of function for this complex construct. There is also research suggesting that rating scales with a small number of response options may be less preferred by respondents (Jones, 1968; Preston & Coleman, 2000), including older adults, who in one study perceived dichotomous scales as being too restrictive (Carp, 1989). Ultimately, the choice of response options will be based on multiple study-specific considerations, including but not limited to the purpose of the measurement and characteristics of the target sample (Lozano et al., 2008).

There were no identical items that appeared as top items in both the low and high concern ranges of interest. Despite this, items that captured the ability to remember appointments—from different questionnaires—appeared in both ranges, suggesting that this specific content warrants inclusion in questionnaires targeting older adults with varying levels of self-perceived cognitive functioning. In addition, prospective memory items (“Remembering what I intended to do”; “Remembering appointments or meetings”) emerged as top items in both self-perceived cognitive functioning ranges. Although most aging studies objectively assess retrospective memory, prospective memory is integral to the maintenance of independent and successful living as it is the cognitive ability that maintains, updates, and monitors our to-do list (Brandimonte et al., 1996). Moreover, failures of prospective memory have important ramifications for older adults’ everyday functioning, emotional well-being, and safety (Kliegel et al., 2016; Mogle et al., 2019; Woods et al., 2012), suggesting a benefit for the inclusion of prospective memory items on questionnaires.

Also, while time referents for top items varied, comparisons to five years ago and the present (including current functioning and the past two weeks) were the most common, and accounted for approximately three quarters of responses, followed by one year ago. Only two items referenced more distant time frames—i.e., high school or college. Based on these results, it may be advisable to avoid items with referents that target the distant past, which could pose difficulties for older adults trying to recall specific instances of remembering or forgetting and lead to inaccurate or biased reports (Cavanaugh et al., 1998; Robinson & Clore, 2002a, 2002b). Another finding was that approximately one third of the top items captured ability, while the majority assessed change. Reporting ability requires appraisal of current status only; reporting change requires appraisal of current and past abilities and contrasting them. The predominance of items that focused on change is consistent with the idea that SCD assesses intraindividual change in cognitive functioning (Jessen et al., 2020).

An important issue is whether our results align with proposed SCD plus features—i.e., characteristics of SCD presumed to be associated with increased risk of cognitive decline (Jessen et al., 2014; 2020). Consistent with the SCD plus framework, we found an overrepresentation of memory items among the top items, compared to other cognitive domains, particularly in the high concern range. Also, while time referents for top items varied, all but two targeted the last five years or a more recent time referent, consistent with the onset of SCD within the last five years. A top item related to worry about memory. This corresponds to the SCD plus feature of worry and with research showing that older adults who express concern or worry about their perceived cognitive difficulties have an increased risk of future objective cognitive decline or dementia (Jessen et al., 2010; St. John & Montgomery, 2002; van Harten et al., 2018; Verfaillie et al., 2019). Although the current study cannot be used to validate the SCD plus concept, results may be used to inform measurement selection in research aimed at refining or expanding SCD plus criteria by suggesting possible items to measure self-perceived cognitive functioning (inclusion of some iteration of the item “I worry about my memory ability”).

Items identified as psychometrically strong should also be considered in relation to previous research and broader trends. Rabin and colleagues (2015) previously identified the 10 most frequently occurring item themes on questionnaires used by participating SCD-I working groups. Items that fell within eight of these previously identified themes (memory change; memory for names of people; general memory problems; losing objects; word finding; remembering appointments; remembering recent conversations; memory for intentions/prospective memory) appeared as top items in the current study. In addition, one of the most frequently occurring self-report items “Compared with one year ago, do you have trouble remembering things?” was identified as a psychometrically poor item in the current analysis. However, a similar item capturing memory change over the same time interval (“Compared with one year ago, do you feel your memory has declined substantially?”) was identified as a psychometrically strong item. These results highlight the need to re-examine items and to consider not only general content and psychometric issues but also nuances in wording and response options when modifying existing questionnaires or designing new ones.

Finally, approximately one-third of our identified top items had stems that would typically be viewed as problematic by virtue of having multiple sub-questions within a single item (Oppenheim, 1992). Some of these items would be considered multi-barreled—i.e., the item asks respondents to rate or respond to two or more different issues while allowing for only one response (Bradburn et al., 2004; Menold, 2020), such as “More trouble recalling names, finding the right word, or completing sentences.” Other top items had stems that inquired about a single aspect of cognition, but were followed by a series of examples, such as “Compared to one year ago, do you have more difficulty managing money (e.g., paying bills, calculating change, completing tax forms)?” Although survey design researchers typically advise against the inclusion of multi-barreled items, which can be confusing or difficult to interpret (DeWalt et al., 2007), they are still frequently used. After analysis of multi-barreled items, Menold (2020) concluded that respondents likely attend to the aspect of the item that is most relevant to them, and that these items did not have inferior reliability, compared to single-barreled questions (Menold, 2020). Additional research would be required to explore how older adults understand and respond to items that are single- versus multi-barreled, and to guide decisions about the inclusion of multi-barreled items in questionnaires targeting this population.

Features of Psychometrically Poor Items

We identified items with poor psychometric properties (Appendix I), which had a few notable features. Some items inquired about issues that may not be highly relevant for most older adults (e.g., “Remembering trivia”). Time referents for some items were very different when compared to those found in psychometrically strong items; two specific time referents, appearing in 6 of 15 items, required ambiguous comparisons (“than you used to be” and “ever before”). Several items referenced very broad or vague areas that may have been difficult to interpret (e.g., “My ability to recall things that happened during my childhood is”; “Are you worried about these complaints”). One had a vague time referent: “Remembering things that happened a long time ago.” Another item was possibly confusing due to the wording: “When I forget to do something I had planned to do, it is usually not because I forgot what I had to do but because I forgot when I had to do it.” Two items came from a measure with a 9-point Likert-type scale, with response options ranging from −4 to +4, which might overwhelm some older adults. Finally, several items inquired about complaints related to the same aspects of cognition that also appear among the top items (language/word finding and attention/concentration), suggesting that older adults may better understand qualifiers such as difficulty, problems, or change (as opposed to complaints) when reporting on their cognitive functioning. Together, these results call attention to the importance of utilizing items with simple, straightforward language, and clear time frames, particularly for older adults (Jobe & Mingay, 1990).

Strengths and Limitations

The current study has notable strengths. The large sample size was undeniably a strength because it diminished sampling variability effects on the population parameter estimates used in the reference study. We included numerous items covering multiple cognitive domains and participants with varying degrees of cognitive concerns. The application of an IRT model to self-perceived cognitive functioning measurement had several advantages. IRT models allow for a variety of data types to be included (Gibbons et al., 2007), and provide an acceptable trait estimate despite the use of non-identical items (Hays et al., 2000). Moreover, IRT can keep item parameters constant even when utilizing different samples (Nguyen et al., 2014) and identify the problematic features of poorly performing items thereby helping researchers avoid these characteristics when developing future items and measures (Hays et al., 2000). IRT is also useful in for implementing CAT, which has the possibility to reduce participant burden during assessments (Bandalos, 2018; Hays et al., 2000). However, items contained in the item bank for the current study cannot be used directly in a CAT without modification due to the vast heterogeneity in important item features (temporal referents, response options, etc.). In addition, although access to and facility with computers is widespread, this may not always be the case in clinical and research settings where diverse older adults are assessed—and fixed-length short forms, which can show strong precision along a trait continuum (Lai et al., 2011) may be useful alternatives.

Several limitations should be noted. Study analyses were specific to the group of items to which we had access, the samples in which they were tested, and the order in which we chose to proceed through our calibration steps. For example, the current study was anchored using the ECog Self measure in the MCSA. Using another measure may have yielded different results and thus, we should consider the current results as preliminary. We chose to evaluate data only from the first occasion that each participant responded to the self-perceived cognitive functioning items. This choice ensured that all data would be independent. Some studies provided multiple waves of data, but others only provided a single wave of data. We could have incorporated all of these data with (much) more complicated models but for this initial work we focused on the simpler one-observation-per-participant data set.

We carried out a small DIF study for the five studies that used the ECog Self (i.e., in a subset of items that were identified as linking items) to assess how the different populations in the different studies would affect the estimation of the item parameters. Importantly, although our results demonstrated that cross-study DIF was present and could be detected at conventional levels of statistical significance, it was responsible for very small impact at the latent trait level. This lends support to the viability of harmonization across studies. Unfortunately, we were not able to carry out DIF testing across different language studies to ensure measurement invariance (or even linguistic equivalence) across different language versions of the questionnaires prior to merging the data. If the same questionnaires were to be used in different studies in different countries or linguistic groups, the approach to DIF analyses would be relatively straightforward. Also, the need for harmonization and co-calibration would be much simpler or unnecessary (in the absence of DIF). However, when different tests or measures are used across studies, to accomplish the harmonization goal we must first identify linking items, which we assume are equivalent (and free from DIF) across country and language. For many cross-study links, the number of linking items was small and not conducive to formal DIF testing.

Relatedly, the geographic and linguistic diversity of our sample, which can be viewed as a strength, also posed challenges in completing the co-calibration, and it placed limits on our ability to use psychometric methods to evaluate the assumptions used to complete the co-calibration activity. In practical terms, while items that were identical or similarly worded (deemed equivalent) may appear well-suited for analyses, subtleties in target languages and cultural differences can lead to participants interpreting and responding differently (Chan et al., 2015). These are important considerations to bear in mind as we use the current results to inform the creation and validation/norming of new self-perceived cognitive functioning questionnaires for use with culturally and linguistically diverse populations. Despite these challenges, we feel that the geographic and linguistic diversity provided inferential strength in the representativeness and generalizability of our results, to the extent that our links were valid. Relatedly, although great effort was made to obtain and include questionnaires and participant data from culturally diverse countries, most of the contributing studies were North American and European; we recognize this as both a limitation and an important future direction. Additionally, diverse study settings were aggregated for the analysis. Given that rates of cognitive concerns are typically lower in population settings (Archer et al., 2015; Jonker et al., 1996), it is possible that conducting the analysis by separating memory clinic and population-based studies may have yielded different results though IRT methods do not require any particular distribution of the underlying trait for calibration.

Finally, we limited our analyses to neuropsychologically intact participants because of the gradual and progressive lack of awareness of cognitive dysfunction that manifests in the middle to late stages of MCI through AD (Galeone et al., 2011; Vogel et al., 2004; Wolfsgruber et al., 2014). We reasoned that the meaning of self-appraisals of cognition in individuals with objective cognitive impairment could be fundamentally different from self-appraisals in those who are neuropsychologically intact—and thus, that separate models might be needed to capture the latent trait in these two populations. By focusing only on neuropsychologically intact individuals, we potentially restricted the range of self-perceived cognitive functioning that could be observed, as reflected by several items with very few responses in the category reflecting the greatest degree of self-perceived cognitive impairment. Nevertheless, there was considerable heterogeneity across neuropsychologically intact older adults in self-perceived cognitive functioning present in each study, including many with significant concerns about how their brains are functioning (while they perform well on objective cognitive tests).

Additional Implications for Scale Construction and Future Directions

An important application of our harmonization effort, and one the field has repeatedly called for, is the utilization of standardized and valid questionnaires (Molinuevo et al., 2017; Röhr et al., 2020). Although our results can be used to inform questionnaire development, it is not possible to create a new questionnaire by simply combining all the items with the smallest SEM within particular ranges. Identification of top items does not imply that if taken together, these items would necessarily result in a reliable and valid questionnaire. Items with psychometric support in the current study necessitate independent validation studies to confirm adequate properties. This validation can use the quality criteria as identified by the Consensus-based Standards for the selection of health Measurement Instruments (COSMIN) approach to quality of measurement (https://www.cosmin.nl), which was developed to standardize the evaluation of clinical outcome assessments. In addition, the top items had widely varying features (temporal referents, response option formats), which would make their combination into a single questionnaire problematic from both content and measurement perspectives. Also, these items may not comprehensively assess the self-perceived cognitive functioning construct and feedback from content experts would be critical. Finally, the unidimensional model may have failed to adequately model non-memory items embedded within questionnaires with predominantly memory items, and future research might revisit and attempt to refine the latent factor using a bifactor model.

In light of these considerations, our immediate next step will be to use our item bank to derive and validate self-perceived cognitive functioning factor scores for existing questionnaires. When validated against relevant outcomes such as objective cognition, biomarkers, and clinical progression, these factor scores will be shared across international studies (Sikkes et al., 2021). After validation of the factor scores, researchers and clinicians from around the globe, who have questionnaire items in common with those from our bank, will be able to derive factor scores for their own participants using our item bank. We also plan to use the current item bank for the construction of targeted questionnaires based on various item features and psychometric considerations. Prior to questionnaire development we plan to carry out focus groups and cognitive interviewing with racial/ethnic and culturally diverse, community-dwelling older adults to enable informed decision making about item content and key aspects of questionnaire structure/design (e.g., temporal referent, number and wording of response options, whether and how to capture change, ability, or both). Also, the input from experts in the SCD field, as well as co-creation with participants experiencing low levels of self-perceived cognitive functioning, is necessary to construct a meaningful questionnaire. Finally, we will refine the new self-perceived cognitive functioning questionnaire(s) based on results of pilot studies, with consideration of reliability and validity evidence.

Conclusions

This study was the first to employ IRT methods to harmonize self-perceived cognitive functioning data across a large number of international studies. In carrying out the analyses, the authors encountered substantial measurement challenges that placed limits on the extent to which assumptions underlying the harmonization could be tested. An implication is that validity tests need to use external variables to further support the use of the resulting item bank. Another immediate future goal is to use the current results to derive screening measures that enable reliable identification of subtle cognitive changes in clinically normal individuals. Such tools can be utilized in multi-study initiatives and may facilitate large-scale screening efforts and targeting of at-risk populations for early intervention and tracking of treatment-related changes.

Figure 1.

Sequence of Data Coding and Analytic Procedures

Figure 2. Overlap Among Self-Perceived Cognitive Functioning Items Across Studies.

Figure 2 displays the overlap between self-perceived cognitive functioning items in the studies (including substudies) and shows how the different SCD-I working group studies and items are linked together. In the top panel, the bars/dots represent items administered within a study. The bottom panel displays additional detail for the portion of the top panel enclosed in a gray rectangle. Overall, the figure reveals very little item overlap across studies and no single item represented across all studies.

Note. The figure displays 27 instead of 24 studies because the EAS Substudy 1, EAS Substudy 2 and Pitt Substudy were linked separately from EAS and University of Pittsburgh/MYHAT studies. Please refer to Table 1 for the definitions of the study abbreviations. Please refer to Table 2 for information about study items and questionnaires.

Figure 3. Item Information Plots.

Figure 3 presents the item information plots for the 601 items included in the analysis. Items presented in blue have the highest information in the ranges of interest (-2 to -1 SDs and 0 to +1 SDs), and these are the items presented in Tables 4a and 4b. All remaining items are illustrated with gray lines.

Key Points.

Question:

Can item response theory be used to link questionnaires of self-perceived cognitive functioning with varied item properties?

Findings:

We harmonized and identified the psychometric properties of 601 self-report items from 40 different questionnaires of 24 aging studies in a secondary data analysis that used an IRT item banking approach.

Importance:

Resulting item scores were on the same metric, facilitating joint or pooled analyses across international aging studies; item data also may be useful to investigators proposing to measure self-perceived cognitive functioning in clinical and research settings.

Next Steps:

Future research will utilize the current results to develop new questionnaires and to determine associations of self-perceived cognitive functioning with relevant outcomes in the context of aging and dementia research.

Appendices

Appendix

Appendix C

Figure 1.

Raincloud Plot of Posterior Predictive P-Values (PPP) for Items the Mayo Clinic Study of Aging (MCSA) from a Unidimensional and Bifactor Model

Figure 2.

Left. Scatterplot of Factor Score Estimates Under Bifactor (y-axis) and Unidimensional Model (x-axis)

Right. Scatterplot of Differences in Factor Score Estimates as a Function of the Average of the Two Factor Scores

Figure 3.

Raincloud Plot of Posterior Predictive P-Values (PPP) for All Items from All Studies from a Unidimensional and Bifactor Model

Appendix D

Number of Studies in which Each of 40 Study Questionnaires was Utilized.

Note. The total number of studies = 24 (substudies are not included); please refer to Table 2 for the definition of questionnaire abbreviations.

Appendix A.

Definitions of ‘Neuropsychologically Intact’ by Participating Studies

Study name	Criteria for defining ‘neuropsychologically intact’
Alzheimer’s Disease Center Clinical Core and Center for Brain Health (ADC-NYU)	No objective evidence of memory deficit on clinical interview. Performed normally on the objective portion of the Brief Cognitive Rating Scale, which includes tests of concentration and calculation, memory, orientation, and functional abilities, carried out during the clinical interview.
Alzheimer’s Disease Neuroimaging Initiative (ADNI)	Normal subjects: CDR score = 0; MMSE range 24–30, delayed recall one 1 paragraph from the Logical Memory II subscale of the Wechsler Memory Scale– Revised (maximum score of 25) with cutoffs as follows: ≥9 for 16 years of education, ≥5 for 8–15 years of education, ≥3 for 0–7 years of education.
Amsterdam Dementia Cohort (Ams Dem Cohort)	No objective impairment as judged by a multidisciplinary team in the university memory clinic, based (amongst others) on the neuropsychological evaluation.
Anti-Amyloid Treatment in Asymptomatic Alzheimer’s Study (A4)	Cognitively unimpaired defined by scores >25 on the MMSE and CDR=0.
Australian Imaging Biomarkers and Lifestyle Flagship Study of Ageing (AIBL)	MMSE scores 27–30; normal performance on Logical Memory; no evidence of significant difficulty on standardized neuropsychological tests of memory, language, attention, executive functioning, and psychomotor speed; CDR =0.
Barcelona Group (Barcelona)	Healthy control participants were volunteers presenting no cognitive decline and normal scores on two screening tests: MMSE and Memory Alteration Test. Also performed in the normal range for age and education on standardized tests of global cognition, verbal and visual episodic memory, visuospatial functioning, language, verbal fluency, working memory, and executive functions.
Bonn Memory Clinic (Bonn)	No cognitive deficit: no score greater than 1.5 SD below age- and education-adjusted norms on at least one subtest of the CERAD-NP battery.
Canadian Longitudinal Study on Aging	Participants were defined as cognitively unimpaired if trained interviewers determined at baseline assessment that they were able to understand the purpose of the study and could provide reliable data. Participants who self-reported at Follow-up 1 that they had received a formal diagnosis of memory problems or dementia by a physician were excluded.
Dartmouth Memory and Aging Study/Indiana Memory and Aging Study (Dart-Indiana)	Objective cognition: scores within 1.5 standard deviations of the mean established for age- and education-matched controls on standardized neuropsychological testing of memory, attention, executive function, language, spatial ability, general intellectual functioning, psychomotor speed, and standard dementia screens.
Einstein Aging Study (EAS) and substudies	No score greater than 1.5 SD below the age-adjusted mean on neuropsychological tests of memory, attention, executive function, visuospatial ability, or language. Tests included: Free and Cued Selective Reminding Test, Logical Memory I from the Wechsler Memory Scale-Revised, Trail Making Test, Digit Span, Letter Fluency, Block Design, Digit Symbol, Category Fluency task (animals, vegetables, and fruits), and the Boston Naming Test.
German Study on Ageing, Cognition, and Dementia in Primary Care Patients (AgeCoDe)	Performed no greater than 1.0 SD below the normative domain scores on the neuropsychological test battery of the Structured Interview for Diagnosis of Dementia of Alzheimer type, Multi-infarct dementia and dementia of other etiology according to DSM-III-R, DSM-IV and ICD-10 (SIDAM).
Harvard Aging Brain Study (HABS)	Cognitively unimpaired: global CDR = 0; MMSE ≥ 27 with educational adjustment (for low education, scores ≥25); and performance in the normal range within education-adjusted norms on the Logical Memory II Delayed Recall Index from the Wechsler Memory Scale.
Health & Aging Brain Study, Health Disparities (HABSHD)	CDR sum of boxes score = 0 and cognitive test scores broadly within normal limits as defined by performance of no more than 1.5 SDs below the mean of the normative range on any neuropsychological test.
Hellenic Longitudinal Investigation of Aging and Diet (HELIAD)	No objective impairment on any cognitive domain (>1.5 SD below age and education expected levels) as part of an extensive neuropsychological battery covering global cognition, verbal and visual memory, visuospatial functioning, language, attention-speed and executive functions.
IMAP Caen Group	Performed in the normal range for age and education on standardized tests of global cognition, verbal and visual episodic memory, visuospatial functioning, language, verbal fluency, working memory, and executive functions.
Indiana Alzheimer’s Disease Center Cohort (IADC)	Without a measurable cognitive deficit.
Leipzig Longitudinal Study of the Aged (LEILA 75+)	Performed no greater than 1.0 SD below the normative domain scores on the neuropsychological test battery of the Structured Interview for Diagnosis of Dementia of Alzheimer type, Multi-infarct dementia and dementia of other etiology according to DSM-III-R, DSM-IV and ICD-10 (SIDAM).
Mayo Clinic Study of Aging (MCSA)	Cognitively unimpaired participants were cognitively and functionally normal as determined by a consensus panel based on clinical and neuropsychological assessment.
Memory Clinic -Fundació ACE (Fundació)	Performed in the normal range for age and education on the standardized Fundació ACE Neuropsychological Battery.
Sydney Memory and Ageing Study (Sydney)	Classified as cognitively normal if performance on all test measures was above the 6.68 percentile (−1.5 SDs) or equivalent score compared to normative published values, controlling for age, sex and education.
University of Pittsburgh Study/ Monongahela- Youghiogheny Health Aging Team (MYHAT); Pittsburgh (Pitt) Substudy	Cognitively normal: composite scores in all domains within 1.0 standard deviation of the mean for the individual’s age-sex- education group. Domains included attention/processing speed, executive function, language, memory, and visuospatial function) – with a composite score for each domain.
Vanderbilt Memory and Alzheimer’s Center (VMAC)	NACC diagnostic criteria: cognitively unimpaired on neuropsychological tests covering 3 cognitive domains, including memory (Logical Memory I and II, NAB List Learning), executive function (Trail Making Test parts A, B, Digits Backward), and language (Animal Fluency Test, Boston Naming Test).
Victoria Subjective Cognitive Decline Study (Victoria SCDS)	Cognitively intact defined as: scored >136 on the DRS-2 Total Score; no score less than -1.5SD on the delayed recall trials of the CVLT-2; scored > 25/30 on the MMSE-2.
Wisconsin Registry for Alzheimer’s Prevention (WRAP)	Cognitively unimpaired determined in one of two ways as described in Johnson et al. (2018): 1) perform within 2SD on all cognitive tests relative to robust norms accounting for age, sex, WRAT-III reading; 2) individuals were flagged for committee review but determined to be unimpaired after considering all available information.

Note. Neuropsychologically intact includes individuals with varying levels of self-perceived cognitive functioning difficulties.

Appendix B.

Sample Equivalent Items

	Measure	Study, Language	Item	Response options	Time frame
Set 1	Multifactorial Memory Questionnaire, MMQ	Bonn Memory Clinic, German	A: Forget a telephone number you use frequently	0=always;	previous two weeks
				1=often;
				2= sometimes;
				3= rarely;
				4= never
	Cognitive Difficulties Scale, CDS-Q	IMAP Caen Group, French	B: Trouble recalling frequently used phone numbers	0=never;	lately
				1=rarely;
				2=sometimes;
				3=often;
				4=very often;
Set 2	Structured Telephone Dementia Assessment, STDA	Harvard Brain Aging Study, English	C: Do you have trouble remembering things from one second to the next?	0=no; 1=yes	current
	VMAC Cognitive Complaint Questionnaire, VMAC CCQ	Vanderbilt Memory and Alzheimer’s Center, English	D: Do you have trouble remembering things from one moment to the next?	0=no; 1=yes	current

Note. Highlighted response options (rarely, never) for items A and B in Set 1 were combined for purposes of data harmonization due to the categories having fewer than five respondents (see Data Recoded). In addition, highlighted response options (often, very often) for item B in Set 1 were combined to establish equivalent response categories (see Item Overlap).

Appendix E.

Number of Linking Items Between Studies

Study	ADC-NYU	ADNI	Ams Dem	A4	AIBL	Barcelona	Bonn	CLSA	Dart-Indiana	EAS	EAS Sub-	EAS Sub-	Age-CoDe	HABS-HD	HABS	HELI-AD	IMAP Caen	IADC	LEILA75+	MCSA	Fundació	Sydney	MYHAT	Pitt Sub-	VMAC	Victoria SCDS	WRAP
ADC-NYU	4	-	-	4	4	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	4	-
ADNI	-	60	20	-	-	-	-	-	59	-	-	20	-	-	39	-	-	60	-	40	-	-	-	-	-	-	-
Ams Dem Cohort	-	20	27	-	-	-	-	-	19	-	-	20	-	-	1	-	-	20	-	-	-	-	-	1	-	-	1
A4	4	-	-	18	5	-	-	-	1	1	1	-	-	1	-	1	-	-	-	-	-	-	-	-	1	6	-
AIBL	4	-	-	5	6	-	-	-	-	-	-	-	-	-	-	-	-	-	1	-	-	1	-	-	1	5	1
Barcelona	-	-	-	-	-	26	1	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	1	-	-
Bonn	-	-	-	-	-	1	32	20	-	-	-	-	-	-	-	-	3	-	-	-	1	-	-	-	-	-	-
CLSA	-	-	-	-	-	-	20	21	-	-	-	1	1	-	-	-	2	-	-	-	1	-	-	-	-	-	-
Dart-Indiana	-	59	19	1	-	-	-	-	173	1	33	19	-	4	39	1	-	59	-	40	-	15	-	-	1	16	-
EAS	-	-	-	1	-	-	-	-	1	17	4	-	-	1	-	1	-	-	-	-	-	-	-	-	2	1	-
EAS Substudy 1	-	-	-	1	-	-	-	-	33	4	121	-	-	1	-	1	-	-	-	-	-	-	-	-	1	33	-
EAS Substudy 2	-	20	20	-	-	-	-	1	19	-	-	42	1	-	-	-	-	20	-	-	-	-	-	-	-	-	-
AgeCoDe	-	-	-	-	-	-	-	1	-	-	-	1	5	1	1	-	-	-	-	-	-	-	-	-	2	-	-
HABS-HD	-	-	-	1	-	-	-	-	4	1	1	-	1	18	1	1	-	-	-	-	-	-	-	-	11	4	-
HABS	-	39	1	-	-	-	-	-	39	-	-	-	1	1	93	-	-	39	-	39	-	-	-	48	2	-	34
HELIAD	-	-	-	1	-	-	-	-	1	1	1	-	-	1	-	15	-	-	-	-	-	-	-	-	2	1	-
IMAP Caen Group	-	-	-	-	-	-	3	3	-	-	-	-	-	-	-	-	32	-	-	-	2	-	-	-	-	-	-
IADC	-	60	20	-	-	-	-	-	59	-	-	20	-	-	39	-	-	60	-	40	-	-	-	-	-	-	-
LEILA75+	-	-	-	-	1	-	-	-	-	-	-	-	-	-	-	-	-	-	1	-	-	1	-	-	1	-	1
MCSA	-	40	-	-	-	-	-	-	40	-	-	-	-	-	39	-	-	40	-	45	-	-	-	-	-	-	-
Fundacio	-	-	-	-	-	-	1	1	-	-	-	-	-	-	-	-	2	-	-	-	28	-	-	-	-	-	-
Sydney	-	-	-	-	1	-	-	-	15	-	-	-	-	-	-	-	-	-	1	-	-	32	-	-	1	-	1
MYHAT																							17	17	-	-	-
Pitt Substudy	-	-	1	-	-	-	-	-	-	-	-	-	-	-	48	-	-	-	-	-	-	-	17	81	-	-	34
VMAC	-	-	-	1	1	1	-	-	1	2	1	-	2	11	2	2	-	-	1	-	-	1	-	-	51	1	1
Victoria SCDS	4	-	-	6	5	-	-	-	16	1	33	-	-	4	-	1	-	-	-	-	-	-	-	-	1	53	-
WRAP	-	-	1	-	1	-	-	-	-	-	-	-	-	-	34	-	-	-	1	-	-	1	-	34	1	-	35
Average Number of Linking Items a	4.0	39.7	11.7	2.3	2.6	1.0	6.3	5.2	21.9	1.6	10.6	13.5	1.2	2.8	24.3	1.1	2.3	40.0	1.0	40.0	1.3	3.8	17.0	25.0	2.0	7.9	10.4

Note. Because substudies are included, the total number of studies = 27; Numbers on the diagonal show the number of items administered by a given study.

^aDoes not include items on the diagonal. The mean number of linking items between studies was 11.1.

Appendix F.

Number of Studies with which Each Study Shares Linking Items

Study Abbreviation	Number of Studies
ADC-NYU	3
ADNI	6
Ams Dem Cohort	7
A4	9
AIBL	7
Barcelona	2
Bonn	4
CLSA	5
Dart-Indiana	14
EAS	7
EAS Substudy 1	7
EAS Substudy 2	5
AgeCoDe	5
HABS-HD	9
HABS	10
HELIAD	7
IMAP Caen Group	3
IADC	6
LEILA75+	4
MCSA	4
Fundació	3
Sydney	5
MYHAT	1
Pitt Substudy	4
VMAC	14
Victoria SCDS	9
WRAP	7
Average	6.2

Note. Because substudies are included, the total number of studies = 27

Appendix G.

Differences in Mean Level of ECog Self Common Latent Variable Means (and standard errors) Relative to the MCSA Under Assumptions of Measurement Invariance (MI) and Partial Measurement Invariance (N = 5,742)

Study	MI	Partial MI	% Difference	Absolute difference
ADNI	−.580 (.040)	−.569 (.042	2	.011
Dart-Indiana	−.329 (.191)	−.317 (.193)	4	.012
HABS	−.103 (.049)	−.121 (.064)	17	.018
IADC	−.171 (.078)	−.139 (.079)	19	.032

Note. Entries under MI report the estimated mean difference (and standard error) for the named study relative to the MCSA from a model that assumes measurement invariance (i.e., no DIF). Entries under Partial MI reported estimated mean differences from a model that allows for some items to have DIF across study.

Appendix H.

15 Items with the Smallest SEM in the Range of -1 to 0

	Questionnaire abbreviation	Cognitive domain	Item stem	Response optionsa	Temporal referent	Ability versus change	SEM; Information
1.	VMAC CCQ	memory	Do you think you have problems with your memory?b	no; yes	current	A	0.32; 9.93
2.	CFI	memory	Compared to one year ago, do you feel that your memory has declined substantially?b	no; maybe; yes	one year ago	C	0.37; 7.45
3.	VMAC CCQ	memory	Do you have problems with your memory compared to the way it was 5 years ago?b	no problems; some minor problems; major problems	5 years ago	C	0.39; 6.53
4.	CFI	memory	Have you been misplacing things more often?d	no; maybe; yes	one year ago	C	0.39; 6.49
5.	CFI	language	Do you have more trouble recalling names, finding the right word, or completing sentences?c	no; maybe; yes	one year ago	C	0.40; 6.26
6.	CFI	memory	Do you find that you are relying more on written reminders (e.g., shopping lists, calendars)?c	no; maybe; yes	one year ago	C	0.42; 5.71
7.	VMAC CCQ	memory	Do you have difficulty remembering a conversation from a few days agob	no; yes	current	A	0.42; 5.69
8.	VMAC CCQ	memory	Overall, do you feel you can remember things as well as you used to?d	no; yes	unclear	C	0.42; 5.65
9.	VMAC CCQ	memory	Do you have complaints about your memory in the last 2 years?b	no; yes	past 2 years	A	0.43; 5.50
10.	ADL Abbrev	memory	Remembering what I intended to dob	much better; slightly better;	5 years ago	C	0.43; 5.37
				no change;
				slightly worse, slightly to moderately worse; much worsea
11.	CFI	executive	Compared to one year ago, do you have more difficulty managing money (e.g., paying bills, calculating change, completing tax forms)?b	no; maybe; yes	one year ago	C	0.43; 5.35
12.	VMAC CCQ	memory	Do you think that your memory is worse than 5 years ago?d	no; yes	5 years ago	C	0.45; 5.00
13.	VMAC CCQ	memory	Do you lose objects more often than you did previously?b	no; yes	unclear	C	0.45; 4.87
14.	CFI	attention	Do you have more trouble following the news, or the plots of books, movies, or TV shows?b	no; maybe; yes	one year ago	C	0.46; 4.78
15.	HELIAD SCI	memory	Do you have difficulty remembering things that you just read or heard?b	no problems; problems	current	A	0.46; 4.70

^aHighlighted response options were combined for purposes of data harmonization due to the categories having fewer than five respondents.

^bItem also found among the top 15 items within the -2 to -1 range (n= 10 in total)

^cItem also found among the top 15 items within the 0 to 1 range (n = 2 in total)

^dItem not found among the top 15 items for either the -2 to -1 range or 0 to 1 range (n = 3 in total)

Note. SEM = Standard Error of Measurement; CFI = Cognitive Function Index; ADL = Activities of Daily Living Rating Scale Self-Version; HELIAD SCI = HELIAD Subjective Cognitive Items; MAC-Q (a) = Memory Complaint Questionnaire; MMQ = Multifactorial Memory Questionnaire; VMAC CCQ = Vanderbilt Memory and Alzheimer’s Center Cognitive Complaint Questionnaire.

Appendix I.

15 Items with the Highest SEM in the Range of −4 to +4

	Questionnaire abbreviation	Cognitive domain	Item stem	Response optionsa	Temporal referent	Ability versus change	SEM; Information
1.	IQCODE – Short	executive	Handling money for shopping	much improved; bit improved;b	compared with 10 years ago	C	73.40; 0.00
				not much change; a bit worse; much worse
2.	CAPM-C	memory	When I forget to do something I had planned to do, it is usually not because I forgot what I had to do but because I forgot when I had to do it	strongly disagree; disagree;	current	A	13.79; 0.01
				agree; strongly agree
3.	Einstein HSA	memory	Compared with one year ago, do you have trouble remembering things?	less often; about the same; more often	compared with one year ago	C	10.22; 0.01
4.	MIA	memory	I am poor at remembering trivia	disagree; strongly disagree;	current	A	8.14; 0.02
				undecided;
				strongly agree agree
5.	Squire	memory	My ability to recall things that happened during my childhood is	better than ever before; almost better than ever before; slightly more of a change a little bit of a change;	“ever before”	C	7.84; 0.02
				no change;
				almost no change; slightly worse than ever before; a little worse than ever before; worse than ever before
6.	Ams Dem Cohort CC	general	Are you worried about these complaints?	no; yes	current	A	7.33; 0.02
7.	SCCS	orientation	Remembering what day/date/month it is?	better; worse; the same;	“than you used to be”	C	6.81; 0.02
8.	MATS	memory	Any other problems with your memory?	no; yes	current	A	6.04; 0.03
9.	Squire	memory	My ability to recall things that happened a long time ago is	better than ever before; almost better than ever before; slightly more of a change; a little bit of a change;	“ever before”	C	5.76; 0.03
				no change;
				almost no change; slightly worse than ever before; a little worse than ever before; worse than ever before
10.	SCCS	memory	Remembering where you’ve put things that you use often? (keys, watch, glasses, etc.)	better; the same; worse	“than you used to be”	C	5.71; 0.03
11.	Ams Dem Cohort CC	language	Do you have complaints about language or difficulty finding the right words?	no; yes	current	A	5.54; 0.03
12.	CFQ	language	Do you find yourself suddenly wondering whether you’ve used a word correctly?	never; very rarely;	in the past 6 months	A	5.43; 0.03
				occasionally; quite often; very often;
13.	Ams Dem Cohort CC	attention	Do you have complaints about your attention and concentration?	no; yes	current	A	5.41; 0.03
14.	SCCS	memory	Remembering things (events, people, etc.) from a long time ago?	better; the same; worse	“than you used to be”	C	5.33; 0.04
15.	SCCS	memory	Finding the right word to use to describe something you know well? (names of familiar objects etc., not names of people)	better; the same; worse	“than you used to be”	C	5.04; 0.04

^aHighlighted response options were combined for purposes of data harmonization due to the categories having fewer than five respondents.

Note. Ams Dem Cohort CC = Amsterdam Dementia Cohort, subjective cognitive concerns screener; CAPM-C = Comprehensive Assessment of Prospective Memory- Section C; CFQ = Cognitive Function Index; Einstein HSA = Albert Einstein Health Self- Assessment; IQCODE – Short = Short Form of the Informant Questionnaire on Cognitive Decline in the Elderly, self-report version; MATS = Memory and Aging Telephone Screen; MIA = Metamemory in Adulthood Questionnaire; SCCS = Subjective Cognitive Complaint Scale-also called the Subjective Memory Scale; Squire = Squire Memory Self-Rating Questionnaire.