Skip to main content
NCBI home page
Journal of Speech, Language, and Hearing Research : JSLHR logoLink to Journal of Speech, Language, and Hearing Research : JSLHR
. 2025 Aug 19;68(9):4358–4375. doi: 10.1044/2025_JSLHR-25-00019

Recognition Versus Ability to Use External Aids: An Exploratory Item Response Analysis of the Functional External Memory Aid Tool

Anna K Saylor a,b,✉, Sanford R Student c, Alyssa M Lanzi a,b
PMCID: PMC12453022  PMID: 40829028

Abstract

Purpose:

Item response theory (IRT) offers an alternative psychometric approach that can provide more specific measurement information beyond classical test theory (CTT). The Functional External Memory Aid Tool (FEMAT) is a performance-based assessment that evaluates everyday strategy use. Developed using CTT, the FEMAT item stems measure various compensation skills such as recognizing when to use a strategy, the ability to use a strategy, and information recall. This study aimed to investigate how the FEMAT aligns with the principles of IRT and whether IRT can provide additional insights about client performance from FEMAT item stems.

Method:

We conducted a preliminary analysis of item responses from the initial FEMAT psychometric evaluation data set (N = 99), screened for IRT assumptions, and ran a partial credit IRT model. To evaluate item stem characteristics, we completed the same IRT evaluation process and then performed analyses of variance to compare item stem difficulty across groupings.

Results:

The overall item analysis revealed a normal distribution of response categories, although some items had low frequencies in Response Categories 1 or 3. The IRT assumptions were met, with evidence of unidimensionality and local independence, and all FEMAT items fit within the acceptable range for the model. Item stems measuring recognition of when to use everyday strategies were significantly more difficult than those assessing the ability to use everyday strategies.

Conclusions:

This IRT analysis expanded the psychometric evidence examining the FEMAT and revealed additional measurement properties of the tool. The item-level detail from the IRT analysis can inform score interpretation and identify potential areas for improvement or refinement of the measure.

Psychometric evaluation of clinical assessment tools is critical for various aspects of health care, from diagnosis to progress monitoring and even policy implementation (Swan et al., 2023). For adults at risk for dementia, clinical assessments are key to identifying a possible earlier clinical stage of cognitive decline known as mild cognitive impairment (MCI). Identifying MCI is crucial to offer behavioral interventions that promote independence and slow cognitive decline (Livingston et al., 2024; Sherman et al., 2017), as adults with MCI are not yet functionally impacted by cognitive impairments (American Psychiatric Association, 2013). Furthermore, assessment tools can be used to develop appropriate and effective treatment plans that promote adherence, as well as measure the effects of an intervention (Swan et al., 2023). This article, which utilizes an assessment tool for allied health professionals working with people with MCI, serves as an example of the need to continuously evaluate assessment tools and the benefit of updated psychometric evaluation to support the tool's intended clinical utility.

According to a recent report by the American Speech-Language-Hearing Association (2023), 70% of surveyed speech-language pathologists (SLPs) identified cognitive communication due to degenerative diseases as a top five clinical service area for adult health care settings. To better understand how SLPs serve this population, Lanzi et al. (2022) surveyed SLPs to learn about their goals, treatment, and assessment practices for adults with MCI or early-stage dementia. The researchers found that 78% of respondents reported writing goals related to compensatory strategies, and 78% reported delivering treatment approaches targeting compensatory strategies (Lanzi et al., 2022). Compensatory cognitive interventions are beneficial for adults with MCI and dementia due to Alzheimer's disease (AD) because, as neural pathways deteriorate over time, these interventions teach alternate means for individuals to access stored information and complete tasks (Lanzi et al., 2017). Furthermore, the strategies learned through compensatory cognitive treatment may help clients in the early stages maintain their ability to complete instrumental activities of daily living (IADLs) independently (Lanzi & Bourgeois, 2020; Lanzi et al., 2021). Specifically, external memory aids (EMAs; e.g., calendar systems) support functional performance in daily tasks for adults with MCI (Greenaway et al., 2013; Kinsella et al., 2015; Lanzi et al., 2018, 2019), and SLPs commonly teach these aids to clients with MCI in clinical practice (Lanzi et al., 2022). To promote long-lasting treatment effects, EMAs must be systematically designed and taught to meet the client's unique needs and preferences (Lanzi et al., 2021; Sohlberg & Turkstra, 2011).

Although various cognitive assessment tools are available, measures are still needed to comprehensively evaluate an individual's use of compensatory strategies, preferences, and needs to support treatment planning. Performance-based assessments can simulate everyday tasks such as writing a check, but they may not account for how clients complete similar tasks in their daily life (with or without compensatory strategies), such as using online banking to pay bills (Schmitter-Edgecombe & Farias, 2018). Patient-reported outcome measures (PROMs) can provide insight into the types and frequency of compensation strategies used (e.g., the Everyday Compensation Questionnaire; Farias et al., 2020). However, PROMs do not provide insight into how clients use the strategies, their preferences, or whether they recognize when to use them in daily tasks. To address these limitations and complement traditional cognitive performance-based measures and PROMs, the Functional External Memory Aid Tool (FEMAT) was developed to support person-centered compensatory cognitive treatment planning (Lanzi et al., 2020, 2023).

Development of the FEMAT

The FEMAT is a performance-based tool that quantifies and describes a person's use of EMAs (e.g., calendars) during simulated IADLs to assess their cognitive abilities, task performance, attempted compensation, and successful use of compensation (Lanzi et al., 2020, 2023). Designed for use with people with MCI, the FEMAT is intended for clinical settings, as it is brief to administer and freely accessible (available for free download on the FEMAT website; Lanzi, n.d.). While the FEMAT is not norm-referenced and should not be used as a sole diagnostic tool, its structure supports person-centered treatment planning by providing insights into a client's everyday strategy use, including their strengths, and identifying areas that may benefit from targeted intervention.

Before administering the FEMAT, the clinician places materials from the EMA test kit on the table in front of the client and says, “I'm going to ask you to role-play through several everyday tasks. Please feel free to use anything on the table or with you to help you complete the tasks” (Lanzi et al., 2020, 2023). The clinician then observes the client discussing, using, or not using an EMA to complete the items that depict everyday tasks. For example, in Item 4, the clinician presents a client with an invitation to an upcoming party and asks them to “show [the clinician] how you would remember the information for the party.” In response, the client may describe using a calendar in their home or physically input the party details into their smartphone calendar. The FEMAT provides clinicians with quantitative data about the client's compensation use during functional tasks (Lanzi et al., 2020, 2023), as well as useful qualitative information about their EMA preferences (e.g., likes using a smartphone app compared to a paper calendar; Lanzi et al., 2023).

The FEMAT was originally developed in English for use in the United States, adhering to rigorous guidelines for item development and content validity (Crocker & Algina, 1986; Lambie et al., 2017). Lanzi et al. (2020) operationalized their working definition of everyday strategy use as “examining one's ability to use EMAs to compensate for impairments in cognition to enhance the completion of everyday tasks,” grounded in the World Health Organization's International Classification of Functioning, Disability and Health framework (World Health Organization, 2001). The initial FEMAT item pool was created by integrating information from multiple sources, including a review of existing compensatory strategy assessment and intervention literature, pilot studies, and discussions with experts and individuals with MCI. The developers followed Kline's (2005) rules for appropriately writing items. The initial draft consisted of 15 items and a 5-point scoring rubric and was evaluated by cognitive communication experts (e.g., dementia researcher, SLP, neuropsychologist, psychometrician). Following expert evaluation and feedback, the FEMAT item bank was reduced to 11 items on a 4-point ordinal scale (i.e., 0–3; Lanzi et al., 2020).

Applying the classical test theory (CTT) framework, Lanzi et al. (2020) determined the 11-item FEMAT demonstrated appropriate psychometric properties for initial validity (e.g., factor structure) and reliability (e.g., internal consistency). Using a sample of 99 older adults with possible MCI, a confirmatory factor analysis specified a three–latent factor model of the FEMAT where standardized factor loadings ranged from (a) medical tasks: .56–.93, (b) IADL tasks: .5–.9, and (c) retrieval tasks: .69–.9. All latent factors were significantly and positively correlated with one another (i.e., r = .52–.71). For internal consistency, Cronbach's alpha (α) was reliable given α = .84, and all item-to-total correlations for each item were adequate (range: .34–.65). For a full description of the FEMAT psychometric development, we direct the readers to Lanzi et al. (2020). For an in-depth FEMAT administration guide, we refer readers to Lanzi et al. (2023).

Classical Test Theory

As noted earlier, the FEMAT was originally evaluated using the CTT framework. CTT is a linear model that posits an observable test score as the sum of two unobservable (i.e., latent) scores: the respondent's true score and the error score. However, this theory has limitations as item difficulty and item discrimination are test dependent. CTT does not formally account for item-level characteristics that may affect an individual's response to a given item (Hambleton & Jones, 1993). For example, item discrimination refers to the degree to which an item can be used to distinguish between individuals with higher and lower levels of knowledge or skill, as reflected by their correct responses (Bandalos, 2018). A set of test items may appear more discriminating when administered to a more heterogeneous sample of people compared to a more homogeneous sample in terms of their knowledge/skill levels. Additionally, the observed score from a CTT-based test (i.e., the sum score) is conditional on the overall test difficulty (Hambleton & Jones, 1993; McNeish & Wolf, 2020).

Item Response Theory

Item response theory (IRT) offers an alternative psychometric approach to account for item-level characteristics in assessments. IRT places items and respondents on a common scale, allowing scoring to account for individual item characteristics (Hambleton & Jones, 1993). IRT has been used to model other SLP performance-based assessments and patient-reported outcomes (Baylor et al., 2013; Fergadiotis et al., 2015, 2023). IRT offers a useful framework to understand information about how a person's score is affected by (a) their ability level of the latent trait (i.e., the construct of interest) and (b) test item characteristics or parameters (e.g., difficulty, discrimination; Thomas, 2011). For example, IRT can inform expectations about how someone with poor verbal naming skills would perform on specific items on a confrontation naming test compared to someone with stronger naming skills. Both might correctly name a common item like “dog,” but the person with lower naming skills would be less likely to correctly name a more difficult item, such as “abacus.” Using an IRT framework can yield detailed information about a person's abilities in relation to specific item difficulty.

In its simplest form, a one-parameter logistic model (i.e., 1PL, Thissen, 1982; or Rasch model, Rasch, 1960), IRT can directly estimate the probability of a client answering an item correctly based on the item difficulty (i.e., the parameter) and the client's ability level (i.e., theta [θ]) of the construct of interest. That is, the probability that a client answering “correctly” (or, in the case of more than two ordered response categories, scoring “higher”) may be affected by either their ability level or how difficult any given item is (Baylor et al., 2011; Fergadiotis et al., 2023; Thomas, 2011). IRT principles may help clinicians determine more detailed information than CTT about a client's ability by directly comparing the difficulty of an item to the client's performance on that item (e.g., at what difficulty level does the client start answering items incorrectly?). This comparison is possible because IRT measures ability and difficulty on the same logistic scale in logits, as often depicted in an item characteristic curve (ICC). In an ICC graph, the x-axis represents the client's ability level (θ) and the y-axis represents the probability of a correct response; comparison of ICCs can provide information about overall item-level difficulty (i.e., an “easier” item would allow a client with “lower” ability to have a high probability to answer the question correctly).

To appropriately use IRT, two main assumptions should be accounted for in the preliminary data screening: correct dimensionality and local independence (Baylor et al., 2011; Fergadiotis et al., 2023; Lamprianou, 2020). Dimensionality refers to the underlying construct(s) being measured by the assessment (e.g., the FEMAT measures “everyday strategy use”). IRT models are equipped to handle both unidimensional (i.e., items assessing one construct of interest) and multidimensional (i.e., items assessing more than one construct) data. Given that many IRT models are premised on measurement of a unidimensional construct, it is important to note that responses to items with a degree of multidimensionality (e.g., item wording, social desirability bias) can quite commonly be treated as statistically unidimensional (Bandalos, 2018). Local independence suggests that items should only be correlated due to their shared relationship with the latent construct; that is, the items should not be related to each other except via the construct of interest (Bandalos, 2018; Baylor et al., 2011; Christensen et al., 2017). Local independence may be violated, for example, when multiple responses use the same stimuli or a response to one item is dependent on the response to another item. Notably, for the 1PL model, an additional assumption is that all items have equal discriminatory value (Fergadiotis et al., 2023; Lamprianou, 2020). Though a full history and review of IRT is beyond the scope of this article, we refer readers to Baylor et al. (2011) for a comprehensive review of IRT and its use in the field of speech-language pathology.

IRT Evaluation of the FEMAT

Although the FEMAT has only been examined at the “test level” using CTT, this tool has unique item-level stem and response constructions that merit further exploration using an IRT framework. The FEMAT items were all designed to assess everyday strategy use, but the specific item stems and response options were written to measure different client skills within the broader framework of “everyday strategy use,” including (a) the ability to use EMAs, (b) the recognition of when to use EMAs, or (c) the recall of relevant information.

For items that assess the client's ability to use EMAs, the stem provides a direct cue to the client to demonstrate how they may use an external aid (i.e., “show me how you would remember … ”). The response options include (0) no response, (1) verbal explanation, (2) attempted to use EMA, and (3) accurately used EMA. Item 4 is an example of an item with an ability stem, “You just received an invitation to an upcoming party. Show me how you would remember the information for the party” (Lanzi et al., 2020, 2023).

For items that assess recognition of using EMAs, the direct cue in the item stem is not provided. However, at the start of the FEMAT administration, clients were told they could use “anything on the table or with [them] to help complete the tasks,” and the FEMAT tool kit (e.g., sticky notes, calendar) is present on the table near them. Response options for recognition items include (0) no attempt to use a strategy, (1) attempted to use a strategy, (2) wrote some of the bold words (key facts), and (3) accurately wrote all bold words. Item 2 is an example of an item with a recognition stem, “I am going to play a voicemail for you and ask you some questions about the voicemail immediately after it stops. The voicemail is about an upcoming appointment” (Lanzi et al., 2020, 2023).

For items that assess recall of information, the stem prompts for the client to recall information previously presented to them (e.g., about the voicemail they previously listened to). The response options include (0) no accurate facts documented, (1) accurately documented one to two facts, (2) accurately documented three to four facts, or (3) accurately documented five facts. Item 3 is an example item with a recall stem, “Now I would like you to fill out this form about facts from the voicemail” (Lanzi et al., 2020, 2023). Notably, a client's performance on recall items may not reflect their attempted everyday strategy use, but rather their cognitive ability (i.e., the client can independently recall more information without the support of EMAs). Given that IRT provides item-level insights, evaluating FEMAT item-level stems could offer useful information about how clients perform on different item types, beyond the overall FEMAT score. For example, SLPs designing compensatory cognitive interventions may want to compare a client's performance on ability-based items and recognition-based items of FEMAT items. This could help identify appropriate starting points for treatment, such as teaching how to use an external aid versus recognizing when to use external aids.

The present study aims to investigate how the FEMAT aligns with the principles of IRT to improve item-level insights. Additionally, it seeks to explore whether IRT can provide differential information about FEMAT item stems (i.e., ability and recognition of everyday strategy use). An exploratory IRT evaluation is necessary to support future psychometric advancements for the FEMAT, such as the development of a short-form and/or computerized adaptive testing within an IRT framework. Accordingly, this article has two main aims:

  1. To assess the alignment of the FEMAT item bank with an IRT framework, including item analysis, evaluation of IRT assumptions, and fit to a partial credit IRT model

  2. To explore whether IRT can offer distinct insights into the measurement of ability and recognition of everyday strategy use within the FEMAT, with implications for developing future tool advancements

Method

Data

Author A.M.L. provided the de-identified data set to author A.K.S., who conducted a post hoc analysis using the data collected in the previous validation study (Lanzi et al., 2020). The FEMAT was initially validated with a sample of 99 primarily White, female older adults with MCI due to possible AD, from the Southeast, Northeast, and Midwest regions of the United States. The sample size was appropriate based on the guidance of N > 50 for polytomous items in a non–high-stakes setting (Linacre, 1994). The data set included both item-level responses and total FEMAT scores.

To examine Aim 2, the FEMAT developer (author A.M.L.) and a member of the FEMAT development team (author A.K.S.) coded each FEMAT item based on its specific item stem (i.e., ability, recognition, recall). Author A.K.S. conducted the preliminary coding of FEMAT items, which was confirmed by expert consensus with author A.M.L. FEMAT Items 1, 4, 7, 8, and 10 were coded as assessing ability; Items 2, 5, and 9 were coded as assessing recognition; and Items 3, 6, and 11 were coded as assessing recall. Recall stem items were not included in Aim 2 because the response scores may be due to the client's cognitive ability (i.e., remembering voicemail details) rather than their everyday strategy use (i.e., writing down voicemail details). The data set examined was collected from a study approved by the University of South Florida Institutional Review Board (No. 00037152), and no further ethical approval was required.

Data Analysis

The study investigated (a) how well the FEMAT item bank aligns with an IRT framework and (b) whether IRT can yield different insights into the measurement of ability and recognition in the use of everyday strategies. In the following sections, we will describe our analysis approach and findings from both aims using these steps: (a) preliminary analysis of item responses, (b) screening for IRT assumptions (i.e., dimensionality, local independence), and (c) partial credit model (PCM; Masters, 1982) IRT analysis. A PCM was used due to the ordered, polytomous response categories of the FEMAT (i.e., 0–3; Bond et al., 2021). All statistical analyses were completed in R (Version 2024.09.1 + 394; R Core Team, 2024); we fitted the PCM via conditional maximum likelihood using the eRm package (Mair & Hatzinger, 2007). Given the small sample, further analysis using a more flexible IRT model, such as the graded response model (Samejima, 1969), is recommended for future investigation.

Preliminary Analysis of Item Responses

An initial analysis of responses provided valuable information about the overall response distribution and how well the FEMAT items function. Item mean scores illustrated relative item difficulty, where lower means indicate more difficult items. Standard deviations identified the magnitude of response spread (e.g., a smaller standard deviation was associated with less response spread and lower differentiation among clients; Bandalos, 2018). Similarly, response frequency distributions revealed which response options were chosen more or less often, indicating how the items differentiated clients of varying ability/trait levels. Item-to-total correlations determined the extent to which an individual item was correlated with the rest of the items (i.e., item homogeneity; Bandalos, 2018). Additionally, we examined the FEMAT's reliability (α) if an item was removed (i.e., “alpha-if-dropped”). For example, if the reliability would increase after removing a specific item, it may suggest that the item is detrimental to the overall assessment (Bandalos, 2018).

IRT Assumptions

Dimensionality. To examine dimensionality, we produced a scree plot visualization and conducted parallel analysis. After confirming the appropriateness for factor analysis, we performed exploratory factor analyses (EFAs) on the FEMAT data. This included using Pearson correlation matrices, the principal axis factoring method, and promax rotation.

Local independence. To assess local independence, we used Yen's (1981) Q3 statistic (i.e., correlating residuals), a common approach in PCM (Christensen et al., 2017). We used a threshold established from previous research indicating a critical value of 0.2 above or below the average correlation to identify potential violations of local independence (Chen & Thissen, 1997; Christensen et al., 2017).

Partial Credit Model

Item fit. Item fit measures how well the predicted responses from the model align with the observed responses (Bandalos, 2018). Two commonly used fit statistics in Rasch modeling are outfit and infit mean square. Outfit (short for “outright” or “outlier-sensitive” fit) quantifies the extent to which an item or person overfits or underfits the model; infit (“information-weighted” fit) is conceptually similar to outfit but weights the data by the information provided by a given item about a given respondent (Lamprianou, 2020). For either statistic, a value of 1 indicates “perfect” fit, with higher values indicating underfit (i.e., noise) and lower values indicating overfit (i.e., more deterministic item response patterns). Although there are no universal guidelines for acceptable fit (Bandalos, 2018), we used infit and outfit mean square values between 0.5 and 1.7 as our cutoff range (Lamprianou, 2020; Wright & Linacre, 1994), which was deemed appropriate given the small sample size of our study (Wu & Adams, 2013).

Item difficulty and person ability. The PCM measures item difficulty/location and person ability using logits (i.e., log-odd units), which provides a consistent scale for directly comparing items and persons. This comparison can be illustrated via a “Wright map” (Bond et al., 2021)—a histogram of person ability estimates combined with a plot of item locations of the latent variable along the x-axis (Lamprianou, 2020). The Wright map illustrates the distribution of person ability levels along the logit scale shared with item difficulty. Additionally, ICCs illustrate the probability of expected responses based on a person's ability level. The x-axis represents the person's latent ability measured in logits, and the y-axis represents the probability of a specific “correct” response (Lamprianou, 2020).

Results

Aim 1: To Assess the Alignment of FEMAT Item Bank With an IRT Framework

Preliminary Analysis of Item Responses

Average response scores ranged from 0.73 to 1.89 (SD range: 0.79–1.23) with skew and kurtosis values within normal limits (i.e., within |2| and |4|, respectively), suggesting responses of items with a relatively normal distribution. Additionally, item analysis yielded moderate-to-strong item–total correlations ranging from r = .49 to r = .72 (Cohen, 1988). FEMAT reliability was calculated at .84, with limited fluctuation if any item was dropped (i.e., “alpha.drop”; ≤ −0.03 point difference). The standard error of measurement was calculated at 0.25. Response frequencies varied across individual items, with some items showing poor distribution across all four response categories (e.g., Item 1 had a 1% frequency in Response Category 1). Relevant item analysis statistics are found in Table 1.

Table 1.

Aim 1 preliminary analysis of item responses.

Item no. M SD Item–total r Alpha if dropped Response frequency
0 1 2 3
1 1.3 1.06 .49 .84 26% 36% 18% 19%
2 0.73 1.03 .72 .82 64% 6% 24% 6%
3 1.39 0.79 .6 .83 11% 46% 34% 8%
4 1.51 0.98 .73 .81 17% 33% 31% 18%
5 0.84 1.09 .67 .82 60% 6% 25% 9%
6 1.54 0.84 .62 .83 9% 41% 36% 13%
7 0.84 0.82 .52 .84 39% 40% 17% 3%
8 1.89 0.92 .57 .83 10% 18% 44% 27%
9 0.82 1.23 .63 .82 68% 1% 13% 18%
10 1.22 1.06 .56 .83 28% 40% 12% 19%
11 1.86 0.89 .71 .82 9% 20% 46% 24%
Open in a new tab

IRT Assumptions

Dimensionality. The Kaiser–Meyer–Olkin Measure of Sampling Adequacy yielded a measure of sampling adequacy equal to 0.80, indicating an EFA was appropriate (i.e., meritorious) for the sample (Kaiser, 1974). Parallel analysis suggested a three-factor solution, though a very strong first factor was indicated through the steep slope from the first to the second factor (see Figure 1). We evaluated one-, two-, and three-factor solutions using Pearson correlation matrices. Figure 2 depicts all factor loading solutions. The one-factor solution explained 33% of the cumulative variance. The two-factor solution explained 42% of the cumulative variance (i.e., 27% and 15%, respectively), and factors were moderately correlated with one another (r = .58). The three-factor solution explained 49% of the cumulative variance (i.e., 18%, 17%, and 15%, respectively); these factors were also moderately correlated with one another (i.e., PA1 × PA2 = .49; PA1 × PA3 = .53; PA2 × PA3 = .44).

Figure 1.

A plot of the Eigen value on the y-axis and the factor number on the x-axis. The actual data points are at (1, 3.6), (2, 0.8), (3, 0.5), (4, 0.2), (5, 0), (6, 0), (7, negative 0.1), (8, negative 0.2), (9, negative 0.3), (and 10, negative 0.3). The simulated data points are at (1, 0.9), (2, 0.5), (3, 0.4), (4, 0.2), (5, 0.1), (6, 0), (7, 0), (8, negative 0.1), (9, negative 0.2), (10, negative 0.25), and (11, negative 0.4).

Open in a new tab

Aim 1 scree plot. This figure illustrates the scree plot for the exploratory factor analysis of 11 Functional External Memory Aid Tool (FEMAT) items. The steep slope of the red line from Factor Numbers 1–2 indicates a strong first factor and provides evidence for unidimensionality of the FEMAT.

Figure 2.

A table lists the principal axis factors for 11 items. The table has 6 columns. The first column, PA1, displays the one factor solution. The second and third columns, PA1, and PA2 display the two factor solutions for the 11 items. The fourth, fifth, and sixth columns, PA1, PA2, and PA3, display the three factor solutions for the 11 items.

Open in a new tab

Aim 1 factor loadings. The first teal column (PA1) indicates the one-factor solution, the middle two unshaded columns (PA1 and PA2) indicate the two-factor solution, and the remaining three teal columns (PA1, PA3, and PA2) indicate the three-factor solution. Each item factor loading for that solution corresponds to the given item's row. PA = principal axis factor.

Local independence. Inclusive of all FEMAT items, Yen's Q3 statistic ranged from −.447 to .378 (average Q3 = −.095), where the minimum cutoff was −.295 and the maximum cutoff was .105. Out of 55 item pairings, 11 of the pairings violated the threshold for local independence, where six pairings fell below the minimum cutoff and five pairings fell above the maximum cutoff. Table 2 describes the pairings that suggest this violation.

Table 2.

Aim 1 violations of local independence.

Item A Item B Q3 Item Stem A – B
#3 #7 −.447 recall – ability
#1 #6 −.370 ability – recall
#6 #10 −.361 recall – ability
#2 #7 −.346 recognition – ability
#1 #11 −.327 ability – recall
#2 #10 −.297 recognition – ability
#6 #11 .110 recall – recall
#5 #6 .148 recognition – recall
#2 #3 .190 recognition – recall
#7 #10 .205 ability – ability
#9 #11 .378 recognition – recall
Open in a new tab

Note. Q3 = Yen's (1981) Q3 statistic.

PCM

Item fit. All FEMAT items fell within the range of acceptable fit statistics. Table 3 describes the fit statistics of each FEMAT item.

Table 3.

Aim 1 fit statistics based on the partial credit model.

FEMAT item no. Mean square outfit Mean square infit
1 1.23 1.27
2 0.53 0.72
3 0.90 0.92
4 0.69 0.71
5 0.78 0.85
6 0.91 0.91
7 0.98 1.00
8 1.04 1.02
9 0.99 0.98
10 1.19 1.12
11 0.71 0.72
Open in a new tab

Note. Fit values closer to 1 indicate more “perfect” fit. FEMAT = Functional External Memory Aid Tool.

Item difficulty and person ability. Item-level difficulty ranged from −0.62 (i.e., Item 8) to 1.12 (i.e., Item 7). Table 4 describes the overall item-level difficulty and the difficulty between each score threshold (e.g., score 1–2 vs. 2–3). For example, Item 4 suggests that a higher response score indicates increased difficulty of the item (i.e., 0–1 = −1.27, 1–2 = 0.01, 2–3 = 0.97). As such, a person with a higher compensation ability would be more likely to receive a higher score on that item compared to someone with a lower compensation ability. Item 5, however, shows a “disordered threshold” where the item is less difficult at a higher threshold (i.e., score 0–1 difficulty = 2.08 vs. score 1–2 difficulty = −1.15). This was a common pattern among all recognition stem items.

Table 4.

Aim 1 item-level information based on the partial credit model.

Item no. Overall item difficulty Threshold 1 (0–1)
 Threshold 2 (1–2)
 Threshold 3 (2–3)
Difficulty Difficulty Difficulty
1 0.13 −0.80 0.75 0.45
2 1.09 2.20 −1.05 2.11
3 0.06 −2.10 0.31 1.96
4 −0.10 −1.27 0.01 0.97
5 0.87 2.08 −1.15 1.69
6 −0.26 −2.29 0.07 1.45
7 1.12 −0.28 1.16 2.47
8 −0.62 −1.44 −1.14 0.73
9 0.70 4.04 −2.27 0.32
10 0.20 −0.78 1.30 0.07
11 −0.61 −1.68 −1.06 0.91
Open in a new tab

Note. A higher value in “Difficulty” indicates more difficult items.

Person ability ranged from −3.77 to 2.2 logits of compensatory ability (M = −0.12, SD = 1.05). The relationship between a person's ability and item difficulty is illustrated via the Wright map (see Figure 3) and ICC for each item (see Figure 4). For example, the ICC plot for Item 1 depicts a high probability that a person with a low compensation ability (−4 logits) will score in the 0 category; that same person has a near 0% probability of scoring in Categories 1–3. However, a person with higher compensation ability (0 logits) has a probability of ~40% to score a 1, ~20% to score a 2, and ~10% to score a 3 on Item 1.

Figure 3.

The image displays a person-item map for the distribution of person abilities and item difficulties on a single scale, measured in logits. At the top, a histogram represents the \

Open in a new tab

Aim 1 Wright map. The Wright map illustrates person ability and Functional External Memory Aid Tool (FEMAT) item-level difficulty along the same logit scale. The top of the figure (“Person Parameter Distribution”) illustrates the distribution of ability across the participant sample. The bottom of the figure depicts the overall item-level difficulty for each FEMAT item (black dots) and item difficulty at each scoring threshold (white dots). The left side of the figure indicates lower person ability and easier items to answer correctly. The right side suggests higher person ability and more difficult items to answer correctly. FEMAT items with asterisks on the right side of the figure indicate the presence of disordered thresholds.

Figure 4.

The image displays a grid of 11 item characteristic curves, ICC, plots. In all plots, the y-axis represents the probability to solve and the x-axis represents the compensation ability in logits. In each plot, 4 curves corresponding to categories 0 to 3 are plotted. The note below the plots is as follows. Plots illustrate the item characteristic curves for each FEMAT item. The x-axis indicates the person's compensation ability in logits, the y-axis represents the probability to solve (or score in) that response category. Response category 0 is the black line, response category 1 is the red line, response category 2 is the green line, and response category 3 is the blue line.

Open in a new tab

Aim 1 item characteristic curve (ICC) plots. Plots illustrate the ICC for each Functional External Memory Aid Tool item. The x-axis indicates the person's compensation ability in logits; the y-axis represents the probability to solve (or score in) that response category. Response Category 0 is the black line, Response Category 1 is the red line, Response Category 2 is the green line, and Response Category 3 is the blue line.

Aim 2: To Explore Whether IRT Can Offer Distinct Insights Into the Measurement of Ability and Recognition of Everyday Strategy Use Within the FEMAT

This section summarizes the results of applying IRT principles to FEMAT stems related to recognition and ability (i.e., Items 1, 2, 4, 5, 7, 8, 9, and 10). Due to the low category response rates in the preliminary item analysis of Aim 1, we combined categories with a response rate of ≤ 6%. Therefore, Item 2 became dichotomous (i.e., score = 0 or 1), and Items 5, 7, and 9 were collapsed into three response categories (i.e., score = 0, 1, or 2). Items 1, 4, 8, and 10 maintained the four response category options (i.e., score = 0, 1, 2, or 3).

Preliminary Analysis of Item Responses

For Item 2, a dichotomous item, the average response was M = 0.3 (SD = 0.46), and for Items 5, 7, and 9 (i.e., response options = 0, 1, or 2), average responses ranged from M = 0.43 to M = 0.81 (SD range: 0.66–0.79). Items 1, 4, 8, and 10 (i.e., response options = 0, 1, 2, or 3) had average responses ranging from M = 1.22 to M = 1.89 (SD range: 0.92–1.06). All skew and kurtosis values were within normal limits (i.e., within |2| and |4|, respectively). Additionally, item analysis indicated moderate-to-strong item–total correlations ranging from r = .55 to r = .73 (Cohen, 1988). FEMAT reliability was calculated at .79, with limited fluctuation if any item was dropped (i.e., “alpha.drop”; ≤ −0.05 point). The standard error of measurement was calculated at 0.21. Response frequencies varied across individual items due to collapsed category responses for this analysis. The relevant item analysis statistics are found in Table 5.

Table 5.

Aim 2 item analysis.

Item no. M SD Item–total r Alpha if dropped Response frequency
0 1 2 3
1 1.3 1.06 .55 .78 26% 36 18% 19%
2 0.30 0.46 .70 .75 70% 30%
4 1.51 0.98 .73 .74 17% 33% 31% 18%
5 0.43 0.66 .63 .76 66% 25% 9%
7 0.81 0.75 .62 .76 39% 40% 20%
8 1.89 0.92 .58 .77 10% 18% 44% 27%
9 0.49 0.79 .61 .77 69% 13% 18%
10 1.22 1.06 .63 .76 28% 40% 12% 19%
Open in a new tab

IRT Assumptions

Dimensionality. The Kaiser–Meyer–Olkin Measure of Sampling Adequacy yielded a measure of sampling adequacy equal to 0.79, indicating EFA was appropriate for the sample (Kaiser, 1974). Parallel analysis suggested a two-factor solution, as also indicated by the scree plot (i.e., the steep slope levels off after 2; see Figure 5). We evaluated one- and two-factor solutions using Pearson correlation matrices. Figure 6 depicts all factor loading solutions. The one-factor solution explained 32% of the cumulative variance. The two-factor solution explained 42% of the cumulative variance (i.e., 22% and 20%, respectively), and factors were moderately correlated with one another (r = .61), though Item 8 did not load onto either factor in the two-factor solution.

Figure 5.

A plot of the Eigen value on the y-axis and the factor number on the x-axis. The actual data points are at (1, 2.6), (2, 0.5), (3, 0.2), (4, 0), (5, 0), (6, negative 0.1), (7, negative 0.2), and (8, negative 0.3). The simulated data points are at (1, 0.7), (2, 0.3), (3, 0.2), (4, 0.1), (5, 0), (6, negative 0.1), (7, negative 0.2), and (8, negative 0.3).

Open in a new tab

Aim 2 scree plot. This figure illustrates the scree plot for the exploratory factor analysis of the recognition and ability Functional External Memory Aid Tool (FEMAT) items. The steep slope of the red line from Factor Numbers 1–2 indicates a strong first factor and provides evidence for unidimensionality of the FEMAT recognition and ability items.

Figure 6.

A table displays the principal axis factors for 10 items. The first column, PA1, indicates the one factor solution for the 10 items. The second and third columns, PA1, and PA2, indicates the two factor solutions for the 10 items.

Open in a new tab

Aim 2 factor loadings. The first teal column (PA1) indicates the one-factor solution, and the remaining two unshaded columns (PA1 and PA2) indicate the two-factor solution. Each item factor loading for that solution corresponds to the given item's row. PA = principal axis factor.

Local independence. Including both recognition and ability FEMAT items, Yen's Q3 statistic ranged from −.336 to .234 (average Q3 = −.128), where the minimum cutoff was −.328 and the maximum cutoff was .072. Out of 28 item pairings, five of the pairings violated the threshold for local independence. Two pairings fell below the minimum cutoff, and three pairings fell above the maximum cutoff. Table 6 shows the pairings that suggested this violation.

Table 6.

Aim 2 violations of local independence.

Item A Item B Q3 Item Stem A – B
#5 #10 −.336 recognition – ability
#2 #7 −.335 recognition – ability
#2 #9 .141 recognition – recognition
#5 #9 .219 recognition – recognition
#2 #5 .234 recognition – recognition
Open in a new tab

Note. Q3 = Yen's (1981) Q3 statistic.

PCM

Item fit. All FEMAT ability and recognition items fell within the range of acceptable fit statistics. Table 7 describes the fit statistics of the recognition and ability items together.

Table 7.

Aim 2 item fit statistics based on the partial credit model.

FEMAT item no. Mean square outfit Mean square infit
1 1.13 1.14
2 0.63 0.76
4 0.70 0.70
5 0.73 0.89
7 0.85 0.90
8 0.99 1.01
9 1.01 1.00
10 0.90 0.93
Open in a new tab

Note. Fit values closer to 1 indicate more “perfect” fit. FEMAT = Functional External Memory Aid Tool.

Item difficulty and person ability. The analyses of combined ability and recognition items yielded item difficulties ranging from −0.63 (i.e., Item 8) to 1.32 (i.e., Item 5). This is inconsistent with the most difficult item determined from the entire FEMAT item bank analysis (i.e., Item 7: ability item for IADL/financial management). We analyzed the ability and recognition items together to determine the relative difficulty of these items to one another; this yielded a difficulty ordering, from easiest to most difficult, of Items 8, 4, 1, 10, 7, 2, 9, and 5. All three recognition stems (i.e., Items 2, 5, and 9) were identified as the most difficult items. Table 8 indicates the overall item-level difficulty and threshold difficulty for ability and recognition stems combined.

Table 8.

Aim 2 item-level information based on the partial credit model.

Item no. Overall item difficulty Threshold 1 (0–1) Threshold 2 (1–2) Threshold 3 (2–3)
1 0.18 −0.82 0.81 0.56
2 0.95 0.95
4 −0.07 −1.33 0.05 1.07
5 1.32 0.99 1.66
7 0.42 −0.26 1.11
8 −0.63 −1.56 −1.15 0.82
9 0.97 1.66 0.28
10 0.25 −0.81 1.37 0.20
Open in a new tab

Note. A higher value in “Difficulty” indicates more difficult items.

For combined ability and recognition stems, person ability ranged from −3.51 to 2.33 logits of compensatory ability (M = −0.10, SD = 1.18). The relationship between person ability and item difficulty is illustrated via the Wright map (see Figure 7) and ICC (see Figure 8).

Figure 7.

The image displays a person-item map for the distribution of person abilities and item difficulties on a single scale, measured in logits. At the top, a histogram represents the \

Open in a new tab

Aim 2 Wright map. The Wright map illustrates person ability and Functional External Memory Aid Tool (FEMAT) ability and recognition item-level difficulty along the same logit scale. The top of the figure (“Person Parameter Distribution”) illustrates the distribution of ability across the participant sample for the subset of FEMAT items. The bottom of the figure depicts the overall item-level difficulty for each ability and recognition FEMAT item (black dots) and item difficulty at each scoring threshold (white dots). The left side of the figure indicates lower person ability and easier items to answer correctly. The right side suggests higher person ability and more difficult items to answer correctly. FEMAT items with asterisks on the right side of the figure indicate the presence of disordered thresholds.

Figure 8.

The image displays a grid of 8 item characteristic curves, ICC, plots. In all plots, the y-axis represents the probability to solve and the x-axis represents the compensation ability in logits. In each plot, curves corresponding to categories 0 to 3 are plotted. The note below the plots is as follows. Plots illustrate the item characteristic curve for each FEMAT item. The x-axis indicates the person's compensation ability in logits, the y-axis represents the probability to solve (or score in) that response category. Response category 0 is the black line, response category 1 is the red line, response category 2 is the green line, and response category 3 is the blue line.

Open in a new tab

Aim 2 item characteristic curve (ICC) plots. Plots illustrate the ICC for each Functional External Memory Aid Tool item. The x-axis indicates the person's compensation ability in logits; the y-axis represents the probability to solve (or score in) that response category. Response Category 0 is the black line, Response Category 1 is the red line, Response Category 2 is the green line, and Response Category 3 is the blue line.

Comparison of Ability and Recognition Items

We conducted factorial analyses of variance (ANOVAs) to compare item difficulty differences between groups of ability and recognition stem items. Results suggested a significant difference in difficulty between ability and recognition stem items at the overall item difficulty location, F(1, 6) = 16.12, p < .01, d = −2.93. Factorial ANOVAs were also calculated for each category difficulty threshold. Though notably, categories were collapsed differently based on response frequencies determined from the preliminary item analysis (i.e., not all items use the same amount of response categories). All ability and recognition stem items included the threshold between 0 and 1, which yielded a statistically significant difference in item difficulty based on item type, F(1, 6) = 38.88, p < .001, d = −2.21. Item 2 was not included in the remaining ANOVA calculations because it was collapsed to be dichotomous (i.e., 0 or 1) and does not contain data for Response Categories 2 or 3. For the threshold between Response Categories 1 and 2, there was no statistically significant difference between ability and recognition stems. Furthermore, all recognition stems were collapsed into scores of 0, 1, or 2 such that no ANOVA between recognition and ability could be calculated at the threshold between scores of 2 and 3.

Discussion

This study investigated how well the FEMAT item bank aligned with an IRT framework and examined whether IRT could provide unique insights into measuring the ability and recognition of everyday strategy use with the FEMAT. The primary findings from Aim 1 suggest that an IRT framework may be appropriate and beneficial for analyzing the FEMAT. Furthermore, the Aim 2 analysis revealed that IRT can help differentiate FEMAT item groups with varying item stems (e.g., items with recognition stems were more difficult than items with ability stems). These findings offer additional insight into the FEMAT's measurement constructs and guidance for future refinements.

Aim 1

The preliminary analysis of item responses revealed that all FEMAT items had a relatively normal distribution of responses across response options. However, a few items had at least one response option with low frequency (i.e., ≤ 6%). Items 2, 5, and 9 had low-frequency responses at a numeric score of 1 corresponding to “attempted to use a strategy … during the task.” Additionally, Items 2 and 7 had low-frequency responses at a numeric score of 3, which represents “accurately wrote/typed all five bold words from the voicemail transcript”/“accurately documented the key components of each bill on a calendar.” Further examination revealed Items 2, 5, and 9 were coded as recognition item stems and shared the same four response options. We hypothesize that Response Option 1 had lower response frequencies than others due to the subtle differences in the response anchors for 1 and 2. Response Option 1 is scored when the client “attempts to use a strategy during the task,” while Response Option 2 is scored when the client “wrote/typed some of the bold phrases [from the stimuli].” The wording between “attempting to use” and “somewhat using” a strategy may lack clarity in distinctly describing the client's performance. To adhere to best practices for scale development and validation, response options should be unambiguous (Boateng et al., 2018). Additionally, the low response frequencies in the “accurately wrote/typed” option for Items 2 and 7 may suggest that certain IADLs pose greater challenges for individuals with MCI (e.g., financial management vs. medication management). This finding aligns with prior research indicating that managing finances (i.e., IADL targeted in FEMAT Item 7) is more impaired in those with MCI compared to controls (e.g., Kim et al., 2009) and may be one of the earliest IADL changes (Gold, 2012).

The analysis of the IRT assumptions suggests that the IRT framework may be suitable for the FEMAT. The strong first factor in the scree plot in Figure 2 provides evidence of unidimensionality, aligning with the FEMAT's assessment of the singular construct of everyday strategy use. However, some item pairings violated the assumption of local independence, indicating that certain items may be related to each other beyond the unidimensional factor. We offer suggestions and insight regarding these violations below. Given the small data set size and overall model stability, readers should consider the term local dependence as an indication that further exploration with a larger data set is warranted, rather than a definitive finding.

While 11 out of 55 item pairings violated the local independence assumption, removing one of the locally dependent items (Edwards et al., 2018) may not be the most meaningful solution. These “locally dependent” item pairings often measured the same item stem (e.g., Items 7 and 10 were locally dependent and are both ability stems), but each item also assessed a unique IADL context (e.g., financial management, schedule management). Measuring a client's ability across different contexts can inform person-centered clinical decision making; any performance differences across contexts (e.g., the client may have more difficulty with medication management than shopping) may guide clinicians to identify priority intervention contexts (MacDonald, 2025).

Three of the item pairs (i.e., Items 2 and 3, 5 and 6, and 9 and 11) suggested local dependence and did not share the same stem. These items share common stimuli and may be considered a “testlet” (i.e., a group of items related to a single content area; Wainer & Kiely, 1987). Specifically, Item 9 is a recognition stem item that states “ … I want you to remind me to tell Amanda to call the insurance company before 5:00,” and Item 11, following a 2-min delay, is a recall stem item where clients are asked, “What did you need to remind me to do?” These items may be locally dependent, as the client's performance on the first item could affect their score on the latter item. However, the success of the latter testlet item may also depend on the client's cognitive ability, rather than solely their everyday strategy use ability, and modeling these items as testlets would not resolve this larger potential issue of construct relevance.

Items 2 and 3, as well as Items 5 and 6, also share common stimuli but do not have the same time delay between stimulus presentation and information retrieval as seen with Items 9 and 11. The pairing of Items 9 and 11 also demonstrates the largest violation of local independence. Compared to the other recall items without a time delay (i.e., Items 3 and 6), Item 11 may rely more on the strategy used for success, while the other recall items' success may be a function of cognitive ability (e.g., short-term memory). Unfortunately, the FEMAT scoring does not distinguish if the client used their strategy to recall information on the second item pair. Future FEMAT work could specify this difference to better differentiate between cognitive ability and everyday strategy use.

The PCM analysis results indicated that all the FEMAT items fell within the acceptable fit range. Notably, the PCM findings revealed a common pattern of disordered thresholds (i.e., higher scores were “easier” to achieve than lower scores) among the recognition stem items, likely due to the low response frequencies for a score of 1 on Items 2, 5, and 9 (i.e., 6%, 6%, and 1% response frequencies, respectively), rather than a disordering of the response options themselves (Adams et al., 2012). The overall appropriate fit among the FEMAT items, along with the exploration of assumptions and item analysis, suggested that IRT may be suitable for future psychometric studies of the FEMAT. Further work is needed to conduct the additional psychometric testing using IRT modeling, and our findings provide insights into potential refinements to the FEMAT.

Aim 2

The preliminary item analysis from Aim 1 revealed low response frequencies for FEMAT items, particularly those with recognition stem items. To address this, we collapsed the response categories of low frequency (i.e., ≤ 6%) to make the response options less ambiguous (i.e., two or three options; did not use a strategy, attempted to use a strategy, accurately used a strategy). Additionally, the PCM for Aim 2 indicated that all items for the model fell within the acceptable range for a clinical observation assessment, further supporting the suitability of IRT in future psychometric evaluations of the FEMAT.

The analysis of IRT assumptions for Aim 2 suggests that a multidimensional IRT model may be appropriate for the FEMAT. Dimensionality analysis revealed that a two-factor solution better fits the item stem structure, potentially distinguishing between the ability and recognition aspects of everyday strategy use. This is further supported by the assessment of local independence, where the two item pairs sharing a common item stem violated the assumption (e.g., Items 7 and 10 are both ability items), indicating that some FEMAT items are connected to the stem beyond the underlying construct. To confirm a multidimensional IRT approach, the FEMAT item bank could be expanded to include more items per stem, as increased test length can improve the accuracy of item parameter estimations (Akour & Al-Omari, 2013; Khan, 2014). Expanding the item bank would also enable the creation of alternate test forms (e.g., for clinical or research use) and the selection of items for treatment based on difficulty and ability estimates (Hula et al., 2020).

The analysis revealed that items with a recognition stem were more challenging than those with an ability stem. Recognition stem items lacked the clinician's prompt to “show me” how to use a strategy. Instead, the client had to self-initiate strategy use to support a future task in the FEMAT, and self-initiation is often not measured in performance-based assessments (Moore et al., 2007). Indeed, successfully remembering future tasks (i.e., prospective memory) is critical for functional independence in daily activities (e.g., remembering to go to appointments; Jones et al., 2021). Early theoretical research suggested that the self-initiation required for prospective memory (e.g., remembering to complete a task) is susceptible to aging (Craik, 1986). However, older adults' performance on prospective memory tasks has been variable across studies (Jones et al., 2021). This variability may be attributed to the nature of the prospective memory tasks, which can be event- or time-based tasks (Einstein & McDaniel, 1990). Event-based prospective tasks are signaled by an external event (e.g., seeing a friend and remembering that you needed to give them a message), whereas time-based events require the person to remember to perform the task after a certain amount of time without any external event or cue (e.g., taking medication at 8:00 p.m.; Einstein & McDaniel, 1990). Given that the FEMAT's recognition-based items relate to this concept of self-initiation in prospective memory, it would be beneficial for future FEMAT development to systematically investigate how recognition to use external aids in task completion may vary by event- and time-based tasks.

Limitations and Future Directions

A major limitation of this preliminary study was that IRT models beyond a unidimensional 1PL model were not explored. Due to the nuanced dimensionality of the FEMAT, as suggested by the IRT assumption results, future studies should evaluate the extent to which other IRT models (e.g., multidimensional, graded response model; Samejima, 1969) may better fit the FEMAT item bank and if more flexible modeling approaches might yield further information about the clinical utility of the tool. A second limitation of the current study is that, since its creation, the FEMAT has been updated to Version 2.0 based on user feedback (Lanzi et al., 2023). In the update, item responses were changed from being specific to what external aids should be used for each item (e.g., Item 1 – “timer” vs. Item 4 – “calendar”) to being more general and consistent across items (e.g., Item 1 and Item 4 – “external aid”). Indeed, this update of more general responses also supports use of a rating scale model (i.e., instead of PCM; Andrich, 1978) that requires fewer parameters to be estimated (see, e.g., Ostini et al., 2015). A rating scale model requires common response options among items, which was updated in Version 2.0, and parameter estimation would be more stable due to the smaller total number of parameters. While the item responses were updated in Version 2.0, the item stems of interest for this analysis (i.e., ability, recognition, and recall) did not vary across the FEMAT versions. The analysis conducted here provides the rationale and direction for continued psychometric work on the updated FEMAT.

Future work on the FEMAT should also consider refining the FEMAT response options. First, in the response options for recall stem items, it may be helpful to specify if the item was answered with or without the use of an external aid to distinguish if performance on those items is due to cognitive ability or everyday strategy ability. Second, future work should be done to reevaluate the number of response options for each item. We observed low response frequencies among certain recognition response options, suggesting they may be able to be collapsed; this may also address the disordered thresholds that were observed in the PCM analyses. Consideration of the response categories may support the overall fit and use of an IRT model for the FEMAT item bank.

Additionally, future FEMAT work should prioritize the tool's validation among a more diverse participant sample. The original validation sample consisted primarily of White females, limiting the generalizability of the findings to that demographic group. Other racial, ethnic, or cultural groups may perceive and use compensatory strategies in unique ways to suit their individual needs and daily activities. This is evidenced in the cultural adaptations of the FEMAT (Saylor et al., 2024) for the United Kingdom, Australia, and Spanish-speaking populations in the United States. Ensuring the FEMAT's clinical applicability for a broad client base is critical, given this diversity.

Conclusions

The FEMAT is a performance-based measure of everyday strategy use for individuals with MCI. It complements other standardized cognitive assessments and PROMs to develop person-centered compensatory cognitive treatment plans. Although the FEMAT demonstrates good reliability under the initial assessment using CTT, our findings suggest that IRT analyses may be more appropriate for the FEMAT with further refinement. IRT can provide additional measurement information about the FEMAT beyond CTT and offer valuable information for clinical applications. Continued research is needed to refine the FEMAT based on insights gained from this exploratory analysis and conduct subsequent IRT-based analyses to enhance the dimensionality of FEMAT information for improved clinical utility.

Author Contributions

Anna K. Saylor: Conceptualization (Equal), Data curation (Supporting), Formal analysis (Lead), Methodology (Equal), Project administration (Lead), Software (Lead), Visualization (Lead), Writing – original draft (Lead), Writing – review & editing (Equal). Sanford R. Student: Conceptualization (Equal), Formal analysis (Supporting), Methodology (Equal), Software (Supporting), Supervision (Supporting), Writing – review & editing (Supporting). Alyssa M. Lanzi: Conceptualization (Supporting), Data curation (Lead), Funding acquisition (Lead), Investigation (Lead), Resources (Lead), Supervision (Lead), Writing – original draft (Supporting), Writing – review & editing (Equal).

Data Availability Statement

Data are available from the corresponding author upon reasonable request.

Acknowledgments

This work was funded by the National Institute on Aging of the National Institutes of Health Grant K23AG070185-01 (awarded to Alyssa M. Lanzi). The authors would like to thank Faith Stagge for her helpful input in the initial phases of this study development. Additionally, the authors would like to thank Michelle Bourgeois for her dedication to student mentorship and her continued legacy in the field.

Funding Statement

This work was funded by the National Institute on Aging of the National Institutes of Health Grant K23AG070185-01 (awarded to Alyssa M. Lanzi).

References

  1. Adams, R. J., Wu, M. L., & Wilson, M. (2012). The Rasch rating model and the disordered threshold controversy. Educational and Psychological Measurement, 72(4), 547–573. 10.1177/0013164411432166 [DOI] [Google Scholar]
  2. Akour, M., & Al-Omari, H. (2013). Empirical investigation of the stability of IRT item-parameters estimation. International Online Journal of Educational Sciences, 5(2), 291–301. https://iojes.net/index.jsp?mod=makale_tr_ozet&makale_id=41119 [Google Scholar]
  3. American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders, fifth edition (DSM-5). American Psychiatric Publishing. 10.1176/appi.books.9780890425596 [DOI] [Google Scholar]
  4. American Speech-Language-Hearing Association. (2023). ASHA 2023 SLP Health Care Survey: Caseload characteristics. https://www.asha.org/siteassets/surveys/2023-slp-hc-survey-caseload-characteristics.pdf [PDF]
  5. Andrich, D. (1978). A rating formulation for ordered response categories. Psychometrika, 43(4), 561–573. 10.1007/BF02293814 [DOI] [Google Scholar]
  6. Bandalos, D. (2018). Measurement theory and applications for the social sciences. Guilford. [Google Scholar]
  7. Baylor, C., Hula, W., Donovan, N. J., Doyle, P. J., Kendall, D., & Yorkston, K. (2011). An introduction to item response theory and Rasch models for speech-language pathologists. American Journal of Speech-Language Pathology, 20(3), 243–259. 10.1044/1058-0360(2011/10-0079) [DOI] [PubMed] [Google Scholar]
  8. Baylor, C., Yorkston, K., Eadie, T., Kim, J., Chung, H., & Amtmann, D. (2013). The Communicative Participation Item Bank (CPIB): Item bank calibration and development of a disorder-generic short form. Journal of Speech, Language, and Hearing Research, 56(4), 1190–1208. 10.1044/1092-4388(2012/12-0140) [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar-Quiñonez, H. R., & Young, S. L. (2018). Best practices for developing and validating scales for health, social, and behavioral research: A primer. Frontiers in Public Health, 6, Article 149. 10.3389/fpubh.2018.00149 [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Bond, T. G., Yan, Z., & Heene, M. (2021). Applying the Rasch model: Fundamental measurement in the human sciences (4th ed.). Routledge. [Google Scholar]
  11. Chen, W.-H., & Thissen, D. (1997). Local dependence indexes for item pairs using item response theory. Journal of Educational and Behavioral Statistics, 22(3), 265–289. 10.3102/10769986022003265 [DOI] [Google Scholar]
  12. Christensen, K. B., Makransky, G., & Horton, M. (2017). Critical values for Yen's Q3: Identification of local dependence in the Rasch model using residual correlations. Applied Psychological Measurement, 41(3), 178–194. 10.1177/0146621616677520 [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Routledge. [Google Scholar]
  14. Craik, F. I. M. (1986). A functional account of age differences in memory. In Klix F. & Hagendorf H. (Eds.), Human memory and cognitive capabilities: Mechanisms and performances (pp. 409–422). Elsevier. [Google Scholar]
  15. Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Holt, Rinehart & Winston. [Google Scholar]
  16. Edwards, M. C., Houts, C. R., & Cai, L. (2018). A diagnostic procedure to detect departures from local independence in item response theory models. Psychological Methods, 23(1), 138–149. 10.1037/met0000121 [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Einstein, G. O., & McDaniel, M. A. (1990). Normal aging and prospective memory. Journal of Experimental Psychology: Learning, Memory, and Cognition, 16(4), 717–726. 10.1037//0278-7393.16.4.717 [DOI] [PubMed] [Google Scholar]
  18. Farias, S. T., Gravano, J., Weakley, A., Schmitter-Edgecombe, M., Harvey, D., Mungas, D., Chan, M., & Giovannetti, T. (2020). The Everyday Compensation (EComp) Questionnaire: Construct validity and associations with diagnosis and longitudinal change in cognition and everyday function in older adults. Journal of the International Neuropsychological Society, 26(3), 303–313. 10.1017/S135561771900119X [DOI] [PubMed] [Google Scholar]
  19. Fergadiotis, G., Casilio, M., Dickey, M. W., Steel, S., Nicholson, H., Fleegle, M., Swiderski, A., & Hula, W. D. (2023). Item response theory modeling of the Verb Naming Test. Journal of Speech, Language, and Hearing Research, 66(5), 1718–1739. 10.1044/2023_JSLHR-22-00458 [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Fergadiotis, G., Kellough, S., & Hula, W. D. (2015). Item response theory modeling of the Philadelphia Naming Test. Journal of Speech, Language, and Hearing Research, 58(3), 865–877. 10.1044/2015_JSLHR-L-14-0249 [DOI] [PubMed] [Google Scholar]
  21. Gold, D. A. (2012). An examination of instrumental activities of daily living assessment in older adults and mild cognitive impairment. Journal of Clinical and Experimental Neuropsychology, 34(1), 11–34. 10.1080/13803395.2011.614598 [DOI] [PubMed] [Google Scholar]
  22. Greenaway, M. C., Duncan, N. L., & Smith, G. E. (2013). The memory support system for mild cognitive impairment: Randomized trial of a cognitive rehabilitation intervention. International Journal of Geriatric Psychiatry, 28(4), 402–409. 10.1002/gps.3838 [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Hambleton, R. K., & Jones, R. W. (1993). An NCME instructional module on comparison of classical test theory and item response theory and their applications to test development. Educational Measurement: Issues and Practices, 12(3), 38–47. 10.1111/j.1745-3992.1993.tb00543.x [DOI] [Google Scholar]
  24. Hula, W. D., Fergadiotis, G., Swiderski, A. M., Silkes, J. P., & Kellough, S. (2020). Empirical evaluation of computer-adaptive alternate short forms for the assessment of anomia severity. Journal of Speech, Language, and Hearing Research, 63(1), 163–172. 10.1044/2019_JSLHR-L-19-0213 [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Jones, W. E., Benge, J. F., & Scullin, M. K. (2021). Preserving prospective memory in daily life: A systematic review and meta-analysis of mnemonic strategy, cognitive training, external memory aid, and combination interventions. Neuropsychology, 35(1), 123–140. 10.1037/neu0000704 [DOI] [PubMed] [Google Scholar]
  26. Kaiser, H. F. (1974). An index of factorial simplicity. Psychometrika, 39(1), 31–36. 10.1007/BF02291575 [DOI] [Google Scholar]
  27. Khan, M. I. (2014). Recovery and stability of item parameter and model fit across varying sample sizes and test lengths in Rasch analysis with small sample. Social Science International, 30(1), 43–60. [Google Scholar]
  28. Kim, K. R., Lee, K. S., Cheong, H.-K., Eom, J.-S., Oh, B. H., & Hong, C. H. (2009). Characteristic profiles of instrumental activities of daily living in different subtypes of mild cognitive impairment. Dementia and Geriatric Cognitive Disorders, 27(3), 278–285. 10.1159/000204765 [DOI] [PubMed] [Google Scholar]
  29. Kinsella, G. J., Ames, D., Storey, E., Ong, B., Pike, K. E., Saling, M. M., Clare, L., Mullaly, E., & Rand, E. (2015). Strategies for improving memory: A randomized trial of memory groups for older people, including those with mild cognitive impairment. Journal of Alzheimer's Disease, 49(1), 31–43. 10.3233/JAD-150378 [DOI] [PubMed] [Google Scholar]
  30. Kline, T. J. B. (2005). Psychological testing: A practical approach to design and evaluation . SAGE. 10.4135/9781483385693 [DOI] [Google Scholar]
  31. Lambie, G. W., Blount, A. J., & Mullen, P. R. (2017). Establishing content-oriented evidence for psychological assessments. Measurement and Evaluation in Counseling and Development, 50(4), 210–216. 10.1080/07481756.2017.1336930 [DOI] [Google Scholar]
  32. Lamprianou, I. (2020). Applying the Rasch model in social sciences using R and BlueSky statistics. Routledge. [Google Scholar]
  33. Lanzi, A. M. (n.d.). Functional External Memory Aid Tool (FEMAT). Resilient Cognitive Aging Lab. https://sites.udel.edu/recall-lab/femat/
  34. Lanzi, A., & Bourgeois, M. S. (2020). Structured External Memory Aid Treatment for mild cognitive impairment. American Journal of Speech-Language Pathology, 29(1S), 474–484. 10.1044/2019_AJSLP-CAC48-18-0209 [DOI] [PubMed] [Google Scholar]
  35. Lanzi, A., Bourgeois, M. S., & Dedrick, R. F. (2020). Development and initial validation of the Functional External Memory Aid Tool. American Journal of Speech-Language Pathology, 29(2), 611–624. 10.1044/2019_AJSLP-19-00075 [DOI] [PubMed] [Google Scholar]
  36. Lanzi, A., Burshnic, V., & Bourgeois, M. S. (2017). Person-centered memory and communication strategies for adults with dementia. Topics in Language Disorders, 37(4), 361–374. 10.1097/TLD.0000000000000136 [DOI] [Google Scholar]
  37. Lanzi, A. M., Saylor, A. K., & Cohen, M. L. (2022). Survey results of speech-language pathologists working with cognitive-communication disorders: Improving practices for mild cognitive impairment and early-stage dementia from Alzheimer's disease. American Journal of Speech-Language Pathology, 31(4), 1653–1671. 10.1044/2022_AJSLP-21-00266 [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Lanzi, A. M., Saylor, A. K., Dedrick, R. F., Bourgeois, M. S., & Cohen, M. L. (2023). The Functional External Memory Aid Tool Version 2.0: A how-to clinical guide. American Journal of Speech-Language Pathology, 32(1), 96–106. 10.1044/2022_AJSLP-22-00178 [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Lanzi, A., Wallace, S., & Bourgeois, M. (2018). External memory aid preferences of individuals with mild memory impairments. Seminars in Speech and Language, 39(03), 211–222. 10.1055/s-0038-1660780 [DOI] [PubMed] [Google Scholar]
  40. Lanzi, A., Wallace, S. E., & Bourgeois, M. (2019). Group external memory aid treatment for mild cognitive impairment. Aphasiology, 33(3), 320–336. 10.1080/02687038.2018.1466104 [DOI] [Google Scholar]
  41. Lanzi, A., Wallace, S. E., Cohen, M. L., & Bourgeois, M. S. (2021). Structured External Memory Aid Treatment (SEMAT) for older adults with mild cognitive impairment: Long-term adherence and acceptability of treatment. Aphasiology, 36(2), 234–250. 10.1080/02687038.2020.1868395 [DOI] [Google Scholar]
  42. Linacre, J. (1994). Sample size and item calibration stability. Rasch Measurement Transactions, 7(4), Article 238. [Google Scholar]
  43. Livingston, G., Huntley, J., Liu, K. Y., Costafreda, S. G., Selbæk, G., Alladi, S., Ames, D., Banerjee, S., Burns, A., Brayne, C., Fox, N. C., Ferri, C. P., Gitlin, L. N., Howard, R., Kales, H. C., Kivimäki, M., Larson, E. B., Nakasujja, N., Rockwood, K., … Mukadam, N. (2024). Dementia prevention, intervention, and care: 2024 report of the Lancet Standing Commission. The Lancet, 404(10452), 572–628. 10.1016/S0140-6736(24)01296-0 [DOI] [PubMed] [Google Scholar]
  44. MacDonald, S. (2025). Practical strategies to optimize cognitive-communication intervention in complex real-world conditions: A life integration approach. American Journal of Speech-Language Pathology, 34(3S), 1615–1641. 10.1044/2024_AJSLP-24-00112 [DOI] [PubMed] [Google Scholar]
  45. Mair, P., & Hatzinger, R. (2007). Extended Rasch modeling: The eRm package for the application of IRT models in R. Journal of Statistical Software, 20(9), 1–20. 10.18637/jss.v020.i09 [DOI] [Google Scholar]
  46. Masters, G. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47(2), 149–174. 10.1007/BF02296272 [DOI] [Google Scholar]
  47. McNeish, D., & Wolf, M. G. (2020). Thinking twice about sum scores. Behavior Research Methods, 52(6), 2287–2305. 10.3758/s13428-020-01398-0 [DOI] [PubMed] [Google Scholar]
  48. Moore, D., Palmer, B., Patterson, T., & Jeste, D. (2007). A review of performance-based measures of functional living skills. Journal of Psychiatric Research, 41(1–2), 97–118. 10.1016/j.jpsychires.2005.10.008 [DOI] [PubMed] [Google Scholar]
  49. Ostini, R., Finkelman, M., & Nering, M. (2015). Selecting among polytomous IRT models. In Reise S. P. & Revicki D. A. (Eds.), Handbook of item response theory modeling: Applications to typical performance assessment (pp. 285–304). Routledge. [Google Scholar]
  50. Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Danmarks Paedagogiske Institut. [Google Scholar]
  51. R Core Team. (2024). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/
  52. Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika, 34(S1), 1–97. 10.1007/BF03372160 [DOI] [Google Scholar]
  53. Saylor, A. K., Appleton, S., Rees, C., Tracey, S., Mendelovits, T., Stagge, F., Cohen, M. L., & Lanzi, A. M. (2024). Cultural adaptation of the Functional External Memory Aid Tool for use in the United Kingdom and Australia. Perspectives of the ASHA Special Interest Groups, 9(3), 868–876. 10.1044/2024_PERSP-23-00261 [DOI] [PMC free article] [PubMed] [Google Scholar]
  54. Schmitter-Edgecombe, M., & Farias, S. T. (2018). Aging and everyday functioning: Measurement, correlates, and future directions. In Smith G. E. & Farias S. T. (Eds.), APA handbook of dementia (pp. 187–217). American Psychological Association. 10.1037/0000076-010 [DOI] [Google Scholar]
  55. Sherman, D. S., Mauser, J., Nuno, M., & Sherzai, D. (2017). The efficacy of cognitive intervention in mild cognitive impairment (MCI): A meta-analysis of outcomes on neuropsychological measures. Neuropsychology Review, 27(4), 440–484. 10.1007/s11065-017-9363-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
  56. Sohlberg, M. M., & Turkstra, L. S. (2011). Optimizing cognitive rehabilitation: Effective instructional methods. Guilford. [Google Scholar]
  57. Swan, K., Speyer, R., Scharitzer, M., Farneti, D., Brown, T., Woisard, V., & Cordier, R. (2023). Measuring what matters in healthcare: A practical guide to psychometric principles and instrument development. Frontiers in Psychology, 14, Article 1225850. 10.3389/fpsyg.2023.1225850 [DOI] [PMC free article] [PubMed] [Google Scholar]
  58. Thissen, D. (1982). Marginal maximum likelihood estimation for the one-parameter logistic model. Psychometrika, 47(2), 175–186. 10.1007/BF02296273 [DOI] [Google Scholar]
  59. Thomas, M. L. (2011). The value of item response theory in clinical assessment: A review. Assessment, 18(3), 291–307. 10.1177/1073191110374797 [DOI] [PubMed] [Google Scholar]
  60. Wainer, H., & Kiely, G. L. (1987). Item clusters and computerized adaptive testing: A case for testlets. Journal of Educational Measurement, 24(3), 185–201. 10.1111/j.1745-3984.1987.tb00274.x [DOI] [Google Scholar]
  61. World Health Organization. (2001). International Classification of Functioning, Disability and Health. https://www.who.int/standards/classifications/international-classification-of-functioning-disability-and-health
  62. Wright, B. D., & Linacre, J. M. (1994). Reasonable mean-square fit values. Rasch Measurement Transactions, 8(3), 370–371. [Google Scholar]
  63. Wu, M., & Adams, R. J. (2013). Properties of Rasch residual fit statistics. Journal of Applied Measurement, 14(4), 339–355. [PubMed] [Google Scholar]
  64. Yen, W. M. (1981). Using simulation results to choose a latent trait model. Applied Psychological Measurement, 5(2), 245–262. 10.1177/014662168100500212 [DOI] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Data Availability Statement

Data are available from the corresponding author upon reasonable request.

Articles from Journal of Speech, Language, and Hearing Research : JSLHR are provided here courtesy of American Speech-Language-Hearing Association

RESOURCES