Development and Reliability of the Prospective Memory Assessment for Children & Youth (PROMACY): A Preliminary Study in a Non-Clinical Sample

Abstract

Prospective memory (PM), “remembering to remember,” has been linked to important functional outcomes in adults. Studies of PM in children and adolescents would benefit from the development and validation of developmentally appropriate clinical measures with known psychometric properties. The Prospective Memory Assessment for Children & Youth (PROMACY), a performance-based measure of PM, was developed for the Pediatric HIV/AIDS Cohort Study Adolescent Master Protocol, Memory and Executive Functioning Substudy, and includes Summary, Time-, and Event-based scores derived from eight trials with an ongoing word search task. Fifty-four healthy perinatally HIV-exposed, uninfected children and youth, mean age 13 years, 54% female, 76% Black/non-Hispanic, and 61% impoverished were included in this psychometric analysis. PROMACY Summary Scores demonstrated low, but broadly acceptable internal consistency as measured by Cronbach’s alpha and Spearman-Brown. Better PROMACY performance was associated with older age, but no other demographic factors. Generally medium-sized correlations were observed between the PROMACY Summary Score and standard clinical measures of retrospective memory, working memory, executive functions, and IQ. Findings from this preliminary psychometric study of non-clinical children and youth provide cautious support for the internal consistency and construct validity of PROMACY’s Summary Score that awaits replication and extension in larger samples of healthy children, youth and clinical populations.

Prospective Memory (PM) describes the complex process of “remembering to remember,” which plays an integral role in the execution of many activities of daily living and health behaviors (e.g., Zogg et al., 2010), including medication adherence (i.e., Poquette et al., 2013; Woods et al., 2008a). PM is vulnerable to a wide range of conditions that affect the central nervous system, such as HIV (e.g., Carey et al., 2006; Harris et al., 2017). PM can be viewed as an umbrella construct, much like executive functions, in that its cognitive architecture is diverse and it is comprised of many different component processes that together exceed the sum of their parts. Although there was early controversy around the uniqueness of PM as a construct (e.g., Cockburn & Smith, 1991), it is now fairly well accepted that PM is separable from retrospective memory and executive functions at the levels of theory (e.g., McDaniel, Umanath, Einstein & Waldum, 2015), measurement (e.g., Gupta et al., 2010), neurobiology (e.g., Woods et al., 2006), and everyday functioning (Kamat et al., 2014) in adults. In other words, PM is reliant upon executive functions and prefrontal systems (Simons, Scholvinck, Gilbert, Frith, & Burgess, 2006; Wiley et al., 1998), as well as retrospective memory and medial temporal networks (Martin et al., 2007), but is nevertheless a unique and separable construct, consideration of which can afford incremental ecological validity (e.g., Woods et al., 2009).

According to widely-cited conceptual models of PM (e.g., Kliegel, Mackinlay, & Jager, 2008; Kliegel, Ropeter, & Mackinley, 2006; McDaniel & Einstein, 2007), “remembering to remember” requires several interrelated processes. These include encoding an intention to be performed in the future (e.g., take medication after dinner) in the presence of a given cue (or time), maintaining that cue – intention pairing in memory over time and during an ongoing activity (e.g., daily routines), monitoring the environment for the cue (or time), detecting and recognizing the cue (or time) when present (e.g., finished dinner), retrieving the intention from retrospective memory, and finally executing the intention as planned (e.g., take medication). PM cues are commonly classified as event-based (e.g., taking medication after dinner) or time-based (e.g., taking medication in an hour). Since event- and time-based PM place different demands on strategic/monitoring processes (Kerns, 2000; Mahy & Moses, 2011; McDaniel & Einstein, 2000) and may require different types of interventions, assessing both event-based and time-based PM is ideal. Yet there are very few clinical instruments available to assess PM, particularly in children and youth.

The Memory for Intentions Screening Test (MIST; Raskin, Buckheit, & Sherrod, 2010) is a standardized measure of PM in adults with strong evidence of construct validity that may provide a viable foundation to develop a validated PM measure for children and youth. The MIST involves performing eight different delayed PM tasks over a 30-minute period, while simultaneously engaged in word-search puzzles that serve as the ongoing task. PM trials are balanced in terms of delay interval (i.e., either a 2-minute or 15-minute delay), response cue (i.e., either Time-Based or Event-Based), and response mode (i.e., verbal or action). Errors are coded according to a detailed scoring system, which operationalizes common errors of omission (e.g., no response) and commission (e.g., task substitution errors). The MIST includes a multiple choice recognition post-test from which a retrieval index is calculated (see Carey et al., 2006). Finally, the MIST includes a 24-hour delayed recall task (e.g., Carey et al., 2006). The MIST has good discriminative (Carey et al., 2006; Woods, Twamley, Dawson, Narvaez, & Jeste, 2007b), convergent/divergent (Bezdicek, Raskin, Altgassen, & Ruzicka, 2014; Carey et al., 2006; Coulehan et al., 2014; Kamat et al., 2014; Raskin, 2009), and ecological validity (Woods et al., 2008b). In adults, better MIST performance is associated with younger age (e.g., Kamat et al., 2014; Raskin et al., 2010), higher educational attainment (e.g., Bezdicek et al., 2014), and gender (e.g., Bezdicek et al., 2014; Palermo et al., 2016). The MIST demonstrates strong inter-rater reliability, internal consistency, and generally satisfactory inter-relationships between the summary score, subscales, and error types (Bezdicek et al., 2014; Woods et al., 2008b). It has played a significant role in our understanding of PM in HIV (e.g., Carey et al., 2006; Poquette et al., 2013; Woods et al., 2008a; Woods et al., 2010), aging (e.g., Kamat et al., 2014), Huntington’s disease (e.g., Nicoll et al., 2014), and Parkinson’s disease (e.g., Raskin et al., 2011) in adults.

Currently, there are no clinical measures for children and adolescents that evaluate both Time- and Event-based PM and possess the strong psychometric properties of the MIST. The literature on PM in children (e.g., Kerns, 2000; Kliegel et al., 2008; Kvavilashvili, Messer, & Ebdon, 2001; Ward, Shum, McKinlay, Baker-Tweney, & Wallace, 2005; Zimmermann & Meier, 2006) supports the need for developmentally appropriate measures of PM. PM is apparent as early as preschool ages and shows developmental progression through childhood, with more advanced PM skills evident in early adolescence (Kliegel et al., 2013; Mahy, Kliegel, & Marcovitch, 2014; Voigt et al., 2014; Yang, Chan, & Shum, 2011). Additionally, an age effect in PM performance has been observed between adolescents and young adults (Zöllig et al., 2007) that may be contingent upon how related (i.e., focal) versus unrelated cues are to ongoing background activities (Wang et al., 2011). There also appear to be links between the development of PM and other executive functions such as inhibition and working memory across childhood (Kliegel et al., 2013; Kretschmer, Voigt, Friedrich, Pfeiffer, & Kliegel, 2014; Mahy et al., 2014; Mäntylä, Carelli, & Forman, 2007).

The scant literature regarding PM in pediatric conditions and diseases suggests PM performance deficits are associated with attention deficit/hyperactivity disorder (Kerns & Price, 2001; Kliegel et al., 2006), traumatic brain injury (McCauley & Levin, 2004; McCauley, McDaniel, Pedroza, Chapman, & Levin, 2009; Ward, Shum, McKinlay, Baker, & Wallace, 2007), sickle cell disease (McCauley & Pedroza, 2010), diabetes mellitus (Osipoff, Dixon, Wilson, & Preston, 2012), juvenile myoclonic epilepsy (Wandschneider et al., 2010), autism (Altgassen, Williams, Bölte, & Kliegel, 2009), and pediatric HIV disease (Harris et al., 2017). Notably, there appear to be virtually no studies on the relationship of PM to other important pediatric outcomes, such as academic performance, socioemotional functioning, or adherence to medical treatment regimens (cf., Sirois et al., 2016). Limitations of prior PM research in children (e.g., see Kerns, 2000) include but are not limited to: “Restricted range of outcome” (p. 63) due to use of single-item or small response sets; the need to balance the amount of information to be recalled (i.e., intentions), with the difficulty and/or relevance of the ongoing distractor task to be interrupted; and, lack of a validated age-appropriate measure. Kerns (2000) also indicated that the ideal PM task should engage the participant from intention formation to execution, be equally motivating to all participants, reduce the risk that the intention is remembered but not executed while simultaneously successfully distract from the to-be-remembered intention, and represent everyday intentions (Kerns, 2000, p. 68).

The paucity of research on PM in children and adolescents may be in part due to the lack of an available, psychometrically validated PM measure specifically designed for use with these age groups. Reliable and valid measurement of PM is a notoriously difficult undertaking, with strong opinions that abound on: the use of symptom-based questionnaires that can be influenced by mood and insight but capture manifest functioning; single-trial naturalistic tasks that have glaring psychometric weaknesses but have ecological relevance; mechanistic experimental measures that may be too narrowly focused to capture the nuances of PM in daily life but enhance our understanding of the construct; and, multifactorial clinical tasks that are not considered “pure” measures of PM but have shaped our understanding of PM and its daily functioning impact in neuropsychological populations (e.g., Uttl & Kibreab, 2011). According to McDaniel & Einstein (2007), the cardinal features of a PM task are that it must: 1) involve the execution of a delayed intention; 2) include an ongoing task, which is sometimes referred to as a distractor task; and, 3) the intention must be executed in a constrained window of time. Thus, it is difficult to develop a task that is brief enough to utilize in clinic, that includes an adequate number of intentions for reliable measurement, and at the same time has sufficiently long delays to dissociate the measure from attention/vigilance (see Woods et al., 2008b). It is our view that the MIST hits the mark in this regard. Unlike many of its experimental counterparts that attempted to isolate PM by including intentions (which by their very nature require encoding and retrieval from retrospective memory) by requiring simple motor (action) responses (e.g., a keyboard press), the MIST developers used clinically rooted cue-intention pairings in order to enhance the MIST’s ecological relevance (e.g., When I show you a request for records form, write your doctor’s name on it). Given the MIST’s inclusion of both Event-based and Time-based components of PM, sound psychometric properties and association with functional outcomes in adults, it is an appropriate instrument to be considered as a basis for the development of a PM measure for use with children and adolescents. Herein we describe the development and validation of the Prospective Memory Assessment for Children & Youth (PROMACY), a measure of PM informed by the MIST and designed for use with children and adolescents ages 8–21.

Two prior studies have used PROMACY in the setting of pediatric HIV disease. Harris et al. (2017) demonstrated that youth with HIV and neurocognitive disorders demonstrated significantly lower PROMACY performance across the Summary, Time-based, and Event-based scales compared to youth with HIV without neurocognitive disorders and to youth without HIV. Age at greatest HIV disease severity also was associated with PROMACY scores in this sample. In a related study of youth with and without HIV, Sirois et al. (2016) found that lower PROMACY scores (i.e., Summary, Time-based, and Event-based) were associated with poorer adaptive behavior, word reading and numerical operations, independent of age and caregiver factors (e.g., education). Thus, while PROMACY shows preliminary evidence of discriminant and ecological validity in pediatric HIV disease, little is known about its psychometric properties in healthy children and adolescents. Evaluating the internal consistency, scale interrelationships, demographic correlates, and convergent validity using data derived from healthy samples is an important next step in understanding the psychometric properties of any new neuropsychological measure intended for use in clinical samples (see Delis, Jacobson, Bondi, Hamilton, & Salmon, 2003).

Method

Participants

Participants were enrolled in the Memory and Executive Functioning substudy (herein referred to as the Memory Study) of the Pediatric HIV/AIDS Cohort Study Adolescent Master Protocol (AMP; https://phacsstudy.org). The parent AMP study is a prospective cohort-based investigation of the long-term effects of perinatal HIV infection and HIV treatments on biomedical and neurobehavioral outcomes in perinatally HIV-infected (PHIV) and perinatally HIV-exposed uninfected (PHEU) children and adolescents conducted at 15 sites in the United States and Puerto Rico. Participants were enrolled in the Memory Study at eight selected urban sites in the United States between 2010 and 2012. Eligibility criteria for the Memory Study included enrollment in the AMP parent study, age 9 to <19 years at entry, ability to participate in testing procedures, and fluency in English (given that some study measures were only available in English).

To examine the properties of PROMACY independent of the effects of HIV and to maximize generalizability of results to healthy child and adolescent populations, only data from the study’s healthy comparison sample (PHEU) were evaluated. The presence of brain injury can potentially influence the psychometrics of any given neuropsychological test (see Delis et al., 2003). As such, of the 75 PHEU participants who completed the Memory Study entry visit, three were excluded from analyses due to intellectual impairment (Full Scale Intelligence Quotient [FSIQ] <70), and 17 youth were excluded due to caregiver reported behavioral impairment on the Behavioral Assessment Systems for Children, Second Edition (Reynolds & Kamphaus, 2004, Behavioral Symptoms Index score ≥70). One additional youth met both intellectual and behavioral exclusions. Analyses reported herein include the remaining 54 healthy PHEU participants who completed PROMACY and the other Memory Study neurocognitive measures, and who represent an optimal seronegative sample within which to examine PROMACY’s psychometric properties (Delis et al., 2003).

Materials and Procedure

Institutional Review Boards at each participating Memory Study site and the Harvard T.H. Chan School of Public Health independently approved this study. Informed consent and youth assent were obtained for all participants according to local institutional guidelines. PROMACY was administered first within the larger Memory Study-specified standardized test battery of PM, retrospective memory, and executive functioning.

Measures

Prospective Memory (PM) – Primary Outcome Measure

Prospective Memory Assessment for Children & Youth (PROMACY)

PROMACY Development and Pilot Testing.

PROMACY, a performance-based test of PM that includes both Event-based and Time-based measures, was developed for the purpose of the Memory Study, adapted from the MIST (Raskin et al., 2010; described above) for use with children and adolescents ages 8–21 years, through agreement with Psychological Assessment Resources, Inc. (PAR, Inc.). To adapt the MIST for children and youth, the study team analyzed the cognitive and memory demands of each aspect of the test and made revisions in several areas. The longest delay interval was shortened from 15 to 10 minutes, thus shortening the overall length of the task from 30 to 20 minutes. Task questions were modified to be more relevant to children (i.e., “a piece of yellow paper” was substituted for “a Request for Records form”). Item vocabulary was reduced to an approximate fourth grade level, and all requests for written responses were simplified (e.g., write your name, number of pets you have). One challenge in developing PROMACY was to make it appropriate for a broad range of children and youth, who have naturally varying vocabulary/reading levels (which of course might be affected in clinical samples). In selecting a Grade 4 reading level, we rationalized that this reading level would provide an ideal balance to ensure sufficient variability in scores. Easier, more engaging word search puzzles also were created and presented in increasing difficulty as needed until the PROMACY task was completed. Finally, the 24-hour delay trial was eliminated. Initial PROMACY modifications were piloted with nine community-based healthy children. Two items were further modified and easier word search puzzles were added.

With these modifications in place, prior to implementation in the Memory Study, PROMACY was further pilot tested with a separate small, diverse group of participants (n = 29), at participating Memory Study sites. Note that, the pilot study participants did not overlap with the validation study sample detailed below. Pilot participants were of mean age 12.1 years (SD = 2.7; range = 8–17 years), 62% male, 48.3% White, non-Hispanic; 17.2% Black, non-Hispanic, and 20.6% Hispanic. Average school grade level was 6.3 (SD = 2.6; range = grades 3–12). Seventeen (58.6%) pilot participants had no reported emotional, behavioral or learning diagnosis, while one or more diagnoses were reported for the remaining 12 pilot participants (41.2%), including ADHD (n = 6; 20.7%), learning disability (n = 4, 13.8%), and emotional disorder and various other conditions (n = 2, 6.9%).

The pilot sample demonstrated a range of performance with mean PROMACY Summary Score = 36.5 (SD = 8.1; range, 18–48) out of 48 possible points. The Summary Score had a modest, though nonsignificant, positive association with age (ρ = .36, p = .16). Overall, the preliminary psychometric properties obtained were encouraging given the small heterogeneous sample from which data were collected. When considering cases without a reported diagnosis (n = 17), the overall Summary Score (M = 37.0; SD = 9.1; range 18–48) demonstrated modest internal consistency (Cronbach’s α = .68) given the small sample size, and respectable split-half reliability (Spearman-Brown r = .82). With these promising preliminary results, it was expected that PROMACY would possess at least adequate basic psychometric properties to measure PM in the proposed larger study.

The Final Study Version of PROMACY.

The final study version of PROMACY contains eight balanced PM items (see Table 1) that include a 2-minute or 10-minute delay, at which time a verbal response (e.g., “In 10 minutes tell me that it is time for a break”) or action (e.g., “When I show you a red pen, write your name on your paper”) is required, prompted by a Time-based cue (e.g., “In 2 minutes ask me when we’re going to be done”) or Event-based cue (e.g., “When I show you the small blue index card, write the name of your school on it”). The participant must remember to perform the specified verbal or action response while engaged in completing word search puzzles as a foreground distractor task. A digital clock was located behind the participant, but available to view at all times during the administration of this task. PROMACY is examiner-administered with items presented in time-specified intervals over a 20-minute period. During administration, the examiner codes each response for accuracy or error type(s) (see below). Immediately following PROMACY completion, an 8-item multiple choice recognition task is administered to assess retrieval (encoding or consolidation) failures, regardless of free-recall performance during PROMACY administration (also see, Woods et al., 2008b). The items on the recognition task include three response options for each trial; one option is the correct instruction given during the trial, while the other two are dummy responses.

Table 1:

Prospective Memory Assessment for Children & Youth (PROMACY): Trial Details and Descriptive Outcomes

Order of Presentation	Intention	Cue	Response Modality	Time Delay (min)	Order of Trial Execution	Cognitive Load	Mean Score	SD	Cronbach α of PROMACY Summary Score with Trial Omitted
Trial 1	Tell me it is time to take a break.	Time	Verbal	10	3	4	1.67	0.67	.54
Trial 2	Write your name on your paper.	Event	Action	10	4	3	1.81	0.48	.63
Trial 3	Ask when we are going to be done.	Time	Verbal	2	1	4	1.37	0.88	.58
Trial 4	Write the names of two animals.	Event	Action	10	5	3	1.72	0.56	.61
Trial 5	Write the name of your school.	Event	Action	2	2	4	1.63	0.65	.55
Trial 6	Write the number of pets you have.	Time	Action	10	8	1	1.93	0.38	.59
Trial 7	Tell me to call home.	Event	Verbal	2	6	2	1.74	0.59	.57
Trial 8	Tell me two things you did this past week.	Time	Verbal	2	7	2	1.11	0.86	.52

^*Note: Min – minutes; Cognitive Load - the number of intentions to be retained until Trial execution; PROMACY – Prospective Memory Assessment for Children & Youth; SD – Standard deviation.

PROMACY’s Event-based and Time-based and two- and 10-minute trials are balanced (four of each) for which both verbal and action responses are required. Note that there is a higher number of action responses on the Time-based scale versus the Event-based scale, which differs from the MIST. Both Event-based and Time-based tasks are scored by assigning two points if the examinee performs the correct response to the correct cue or at the correct time; one point if s/he performs an incorrect response to the correct cue or at the correct time, or performs the correct response to an incorrect cue or at the incorrect time, or recognizes the cue/time but states (or otherwise indicates) s/he does not remember what to do; and, zero points if the examinee performs an incorrect response to an incorrect cue or at the incorrect time, or provides no response at all.

Error Types are coded as Omission/no response (no response provided); Task Substitution (incorrect response for an Event-based cue, or incorrect response given at the correct time for a Time-based cue); Loss of Content (recognition of the cue or recognition of the appropriate time to respond, but no recollection of the correct response itself, as indicated by the participant shaking his or her head “no,” or stating, “I forgot”); Loss of Time (correct response performed at the wrong time, with discrepancy greater than +/− 1 minute for 2-minute cues and greater than +/− 2 minutes for 10-minute cues); Place Losing Omission (only part of the task is performed because either the participant does not recall the entire task or becomes distracted prior to task completion); Random (errors that do not fit any of the other categories).

PROMACY includes six subscales based on cue type (i.e., Time- or Event-based, 2-minute or 10-minute) or response modality (i.e., action or verbal responses). Subscale scores range from 0–8 points and are derived by summing the four items that comprise each subscale (see Table 1). The PROMACY Summary Score ranges from 0–48 points and is calculated by summing the six subscale totals.

Neuropsychological Battery

The following battery of tests was administered in a specified order following completion of the PROMACY task.

Naturalistic Event-Based Prospective Memory Task (NEPT; McCauley et al., 2009; McCauley et al., 2011):

The NEPT is a measure of naturalistic PM previously used in studies of children with sickle cell disease (McCauley & Pedroza, 2010) and traumatic brain injury (McCauley et al., 2009; 2011). The NEPT task was embedded within a structured battery of standardized memory, executive function, attention and processing speed measures. For this task, the child is instructed to respond with, “Please give me three points,” each time the examiner gives the verbal cue, “Let’s try something different.” Three presentations of the verbal cue are provided at predetermined points during the ongoing testing tasks at approximately 15-minute intervals; the participant is to respond within five seconds of the cue presentation to be awarded points (although the participant is instructed to request three points, for scoring purposes, four points are assigned for each correct response). An Event-based score, a Retrospective Memory score and an Intention total score are obtained. The Event-based score (range 0–12 points) was the primary NEPT outcome of interest in relation to PROMACY.

Wide Range Assessment of Memory and Learning – 2nd Edition (WRAML2; Sheslow & Adams, 2003).

Two WRAML2 subtests were administered, Verbal Learning and Design Memory (including immediate recall, delayed recall, and recognition trials).

Wechsler Intelligence Scale for Children – 4^th Edition (WISC-IV; Wechsler, 2003) or Wechsler Adult Intelligence Scale, 4^th Edition (WAIS-IV; Wechsler, 2008).

Participants completed the age-appropriate version of the Digit Span and Coding subtests as proxies for working memory and processing speed, respectively, for the Memory Study. Within the parent study, all participants were administered subtests to obtain an FSIQ, and Working Memory Index (WMI), which were abstracted for these analyses (see Smith, et al., 2012).

Delis Kaplan Executive Function System (DKEFS; Delis, Kaplan, & Kramer, 2001).

Four DKEFS subtests were included as validation instruments with PROMACY: Verbal Fluency, Design Fluency, Color-Word Interference, and 20 Questions (ref. Strauss, Sherman, & Spreen, 2006).

Other Variables of Interest

Child and Caregiver Demographic Information.

Demographic information was collected in the parent study via interview with the primary caregiver and abstracted for these analyses, including: Child age, sex, race, ethnicity, and primary language, household income, caregiver relationship to the child, and caregiver education.

Statistical Analyses

PROMACY scores were summarized using descriptive statistics, and compared across subgroups defined by child socio-demographic characteristics, including sex (male/female), age group (9-<12, 12-<15, 15-<18, 18–19 years), race (Black/non-Black), and ethnicity (Hispanic/non-Hispanic) using Wilcoxon rank-sum tests. Frequency of error scores for various types of errors and total errors were tabulated. Internal consistency of PROMACY was evaluated using standardized measures of Spearman-Brown correlations and Cronbach’s alpha. Cronbach’s alpha coefficients were computed for the Summary Score, which included all 8 trials, and for each subscale score (Time-based, Event-based, Verbal response, Action response, 2-minute, 10-minute). Additionally, Cronbach’s alpha coefficients were computed for the Summary Score, deleting each respective individual trial from the computation demonstrating the relative contribution of each trial to the reliability of the Summary Score. Classically, internal consistency values above 0.7 are interpreted as acceptable, with values < 0.5 being unacceptable, although – as with critical alpha levels – there is much debate regarding the reasonable exceptions to these classic rules (Bhatnagar, Kim, & Many, 2014; George & Mallery, 2002). Due to skewed distribution of PROMACY scores, Spearman correlations were used to evaluate PROMACY scores against other measures of retrospective memory, executive functioning, and aspects of cognitive functioning to assess preliminary convergent construct validity. Analyses were conducted using SAS version 9.2 (SAS Institute, Cary, NC).

Results

Included in analyses were 54 healthy youth without cognitive impairment or caregiver-reported behavioral impairment of mean age 13 years, who were 54% female, 76% Black, 24% Hispanic, with 61% living in poverty (annual household income <$20,000/year). Overall, 74% of caregivers had completed high school. Additional participant/caregiver characteristics are presented in Table 2.

Table 2:

PROMACY Child Demographic and Caregiver Characteristics

Characteristic	Category	Mean (SD) or n (%)
Youth Age at PROMACY		13.0 (2.6)
Age Group	9-<12	20 (37%)
	12-<15	22 (41%)
	15-<18	10 (19%)
	18-19	2 (4%)
Gender	Male	25 (46%)
	Female	29 (54%)
Black Race		41 (76%)
Hispanic Ethnicity		13 (24%)
Household Income	$0- 20,000	33 (61%)
	$20,001- 40,000	13 (24%)
	≥$40,001	8 (15%)
Caregiver is H.S. Graduate		40 (74%)

(n = 54). SD – Standard deviation; PROMACY – Prospective Memory Assessment for Children & Youth; H.S. – high school.

Summary statistics for PROMACY trials, the Summary Score and subscales are presented in Table 1 and 3, respectively. A series of Spearman-Brown correlations were computed to evaluate PROMACY’s internal consistency. The Spearman-Brown adjusted correlation for PROMACY (i.e., split-half reliability, even versus odd test items) was .67. Cronbach’s α coefficient for PROMACY’s Summary Score was .60. Cronbach’s α coefficients for the six individual PROMACY subscale scores tended to be lower than that of the Summary Score, ranging from α = .22-.64, with only the Time-based subscale (α = .64) rising above .6 (see Table 3). Cronbach’s α coefficients for the PROMACY Summary Score with deletion of each respective Trial ranged from α = .52 - .63 demonstrating the relative contribution of each trial to the reliability of the Summary Score as shown in Table 1. Cronbach’s α coefficients for the PROMACY Summary Score were also calculated within levels of each demographic variable and approached acceptable values for older youth (≥ 15 years, α = .58) and female gender (α = .57), and demonstrated acceptable values for Black race (α = .62), non-Hispanic ethnicity (α = .63), and male gender (α = .63). There were no significant differences in mean PROMACY Summary Score by levels of any demographic variable other than age. Older youth (≥ 15 years) scored higher on the PROMACY Summary Score than younger (< 15 years) participants (M = 44.5 vs. 37.4, respectively, p < .009).

Table 3.

Summary Statistics and Cronbach Alpha Coefficients for PROMACY Scores

PROMACY Score	Possible Score Range	Mean (SD)	Median	Interquartile Range (IQR) (25th, 75th percentiles)	Standardized Cronbach Alpha
Summary	0-48	38.94 (8.13)	42.0	(33, 45)	.60
Time-Based	0-8	5.89 (2.09)	6.5	(5, 8)	.64
Event-Based	0-8	7.09 (1.32)	8.0	(6, 8)	.47
Verbal	0-8	6.70 (1.63)	7.0	(6, 8)	.46
Action	0-8	6.28 (1.57)	6.0	(5, 8)	.42
2-minute	0-8	7.15 (1.20)	8.0	(6, 8)	.22
10-minute	0-8	5.83 (1.85)	6.0	(5, 7)	.45
Total Errors	0-6	2.06 (1.62)	2.0	(1, 3)	---
Total Recognition	0-8	7.87 (0.34)	8.0	(8, 8)	---
Word Search	0-105	39.98 (13.02)	37.0	(33, 48)	---

PROMACY – Prospective Memory Assessment for Children &Youth; SD – Standard deviation.

In evaluating PROMACY error scores, the most common error type was Omission (M = 0.96, n = 27) followed by Task Substitution (M = 0.61, n = 24) and Loss of Content errors (M = 0.39, n = 17). Loss of Time (n = 3) and Place Losing (n = 1) were much less frequently observed and Random errors did not occur in this healthy sample. The mean number of errors was 2. One-fifth of participants did not commit any errors, 24% committed only one error, 17% two errors, 15% three errors, and 24% ≥ four errors. The mean Recognition post-task score was 7.87 (out of 8), demonstrating that most participants recognized the required PM tasks after the fact, regardless of task performance.

Correlations of PROMACY with other Neuropsychological Measures:

Correlations of neurocognitive measures with the PROMACY Summary Score are reported in Table 4. The following tests/subtests demonstrated significant correlations with the PROMACY Summary Score: WRAML2 Verbal Learning Immediate and Delayed Recall, WISC-IV/WAIS-IV Working Memory Index and FSIQ, and the Letter Fluency and Category Fluency, Color-Word Interference Word Reading, and 20 Questions Initial Abstraction Score from the DKEFS. For these significant results the Spearman correlations ranged from ρ = .30 to .53 (M = .41). In a linear regression model considering all of these correlated assessments, only the WISC-IV/WAIS-IV Working Memory Index and the DKEFS Category Fluency score were significantly associated with the PROMACY Summary Score after adjustment for age. There was no evidence of collinearity in regression models, with no indication of instability of parameter estimates and variance inflation factors close to 1 for all predictors. PROMACY was not associated with the NEPT (p > .10). Correlations between the standard clinical measures and the Time-based scales of PROMACY are provided in Table 4 for descriptive purposes. Correlations with Event-based scales are not reported due to low reliability.

Table 4:

Spearman Correlation Coefficients of PROMACY Scores with Other Neuropsychological Measures

Construct (Test/Subtest)		Subscale Mean(SD)	PROMACY Summary Score Correlation	PROMACY TB Score Correlation
Prospective Memory (NEPT)
	Event-based score	6.74 (4.96)	−.04	−.10
Retrospective Memory (WRAML2)
	Verbal Learning Total	9.26 (2.46)	.39**	.30
	Verbal Learning Recall	9.68 (2.93)	.38**	.38
	Design Memory Total	8.64 (2.96)	.27	.28
	Design Memory Recognition	8.94 (3.38)	.17	.24
Working Memory (WISC-IV/WAIS-IV)
	Working Memory Index (n=50)	94.16 (11.83)	.53**	.44
Processing Speed (WISC-IV/WAIS-IV)
	Coding	7.78 (2.96)	.19	.22
Executive Function (DKEFS)
	Verbal Fluency
	Letter Fluency	8.72 (2.57)	.46**	.48
	Category Fluency	10.06 (2.99)	.42**	.36
	Category Switching	9.11 (2.75)	.20	.24
	Design Fluency
	Filled Dots	8.37 (2.11)	.18	.14
	Empty Dots	8.67 (2.22)	.20	.22
	Switching	9.11 (2.55)	−.08	−.10
Inhibition/Interference (DKEFS Color-Word Interference)
	Color Naming	8.13 (3.57)	.20	.10
	Word Reading	9.54 (3.44)	.37**	.29
	Inhibition	8.64 (2.89)	.14	.14
	Inhibition/Switching	8.44 (3.25)	.15	.09
Problem-Solving/Concept Formation (DKEFS 20 Questions)
	Abstraction	8.38 (2.76)	.30**	.27
FSIQ
	WISC-IV/WAIS-IV FSIQ (n=50)	92.24 (11.22)	.45**	.42

^*p < .05;

^**p < .01

(n = 54).PROMACY – Prospective Memory Assessment for Children & Youth; SD – Standard Deviation; EB – Event-based; TB – Time-based; NEPT – Naturalistic Event-Based Prospective Memory Task; WRAML2 – Wide Range Assessment of Memory & Learning, 2nd Edition; DKEFS – Delis-Kaplan Executive Function System; WISC-IV – Wechsler Intelligence Scale for Children, 4th Edition; WAIS-IV – Wechsler Adult Intelligence Scale, 4th Edition; FSIQ – Full Scale Intelligence Quotient. Note: Significance testing was not performed on the Time--based correlations, which are provided only for descriptive purposes. Event-based correlations are not reported due to low reliability.

Discussion

PM is a unique construct that has tremendous ecological relevance; however, there is currently a lack of validated, psychometrically sound PM measures for use with children and adolescents. Here we describe the development and initial psychometric properties of PROMACY, which is a new PM measure that was adapted from a well-validated adult task (i.e., MIST; Raskin et al., 2010). Recent data from our colleagues provided preliminary support for the discriminant (Harris et al., 2017) and ecological (Sirois et al., 2016) validity of PROMACY in the context of pediatric HIV disease. In the current study of 54 healthy children and adolescents without HIV infection or cognitive or behavioral impairment, PROMACY demonstrates both strengths and limitations as a measure of PM. At face value, examiner observation revealed that the level of engagement, motivation and effort put forth by participants demonstrated PROMACY is acceptable for children, adolescents, and youth regardless of age. That is, youth appropriately engaged in completing the distractor task word search puzzles, with older youth completing more word searches than younger participants.

In terms of reliability, the PROMACY Summary Score had low but acceptable internal consistency. By traditional standards, Summary Score reliability values < .7 would be considered questionable. However, one must keep in mind the necessary brevity of any clinical PM task, which demands sufficiently long intervals between targets to distinguish the construct from other aspects of cognition (e.g., vigilance), and thereby reduces the number of available items. For cognitive tasks with fewer items, the observed reliability values would be considered “acceptable” (e.g., Bhatnagar, Kim, & Many, 2014; George & Mallery, 2002) and are consistent with adult studies on the MIST (e.g., Bezdicek et al., 2014; Woods et al., 2008). The MIST, for example, has a Spearman-Brown coefficient of .7 in a small sample of healthy adults. The PROMACY Summary Score approached acceptable internal consistency reliability for older youth (≥ 15 years) and female gender, and demonstrated acceptable internal consistency reliability for Black race, non-Hispanic ethnicity and male gender. Aside from the Time-based subscale score, no other PROMACY subscale demonstrated acceptable reliability, even according to standards for shorter measures. It is possible that such low reliability coefficients reflect the instability of the task or the restricted range of scores that are sometimes derived from healthy samples, as has been shown in the adult MIST literature (e.g., Raskin et al., 2011). When individual trials were removed from the Cronbach’s α calculation for the Summary Score, no Cronbach’s α coefficient was below .52, suggesting each trial meaningfully contributes to the internal consistency of the Summary Score, despite relatively restricted range of scores.

PROMACY’s Event-based scale showed very limited reliability and notable ceiling effects that clearly limit its usefulness. One possible contributor to these psychometric problems is the lower item grade level, which may have been too easy for the older youth in this study (as evidenced in the strong ceiling effects in that subgroup). Furthermore, it is possible that the unequal distribution of the action and verbal responses on the Event- versus Time-based scales may have contributed to the differences in reliabilities and ceiling effects on these scales. However, the Event-based scale had more verbal responses, which might actually be expected to be associated with lower performance (e.g., Cohen, 1989). Thus it will be important for future studies to follow the recommendations of Delis et al. (2003) to examine the internal consistency and test-retest reliability of PROMACY in a wide range of clinical samples, which may provide a larger range of scores, particularly on Event-based trials. Future studies may also wish to increase the difficulty level of PROMACY’s Event-based trials, for example by reducing their semantic relatedness (e.g., Woods et al., 2010).

Of course, it is possible for a shorter neuropsychological test such as PROMACY (i.e., with only 8 items) to demonstrate modest internal consistency, but still be clinically useful in discriminating clinical populations, detecting everyday functioning difficulties, and relating to other neurocognitive constructs (see Loevinger, 1954). In this regard, prior studies show that PROMACY differentiates youth with HIV and neurocognitive disorders from youth with HIV without cognitive disorders and from youth without HIV, independent of relevant clinicodemographic co-factors (Harris et al., 2017). Moreover, PROMACY is an independent predictor of relevant functional outcomes, including adaptive behavior and academic performance (Sirois et al., 2016). The current study also showed that the PROMACY Summary Score demonstrated moderate associations (i.e., convergent validity) with well-validated neurocognitive tests, including measures of verbal learning and recall, working memory, Full Scale IQ, letter and category fluency, word reading, and problem solving (see Table 4). In general, verbally-based neurocognitive tests demonstrated larger associations with PROMACY Summary Scores than non-verbal tests. Specifically, working memory and category fluency emerged as the strongest predictors of the PROMACY Summary Score while Full Scale IQ was the weakest predictor. The direction and magnitude of these associations were highly consistent with findings from psychometric studies of the MIST in middle-aged (e.g., Carey et al., 2006) and older (e.g., Bezdicek et al., 2014; Kamat et al., 2014) healthy adults, as well as in some clinical samples (e.g., Carey et al., 2006; Coulehan et al., 2014). Such data fit with current conceptual models regarding the cognitive demands of PM, particularly in regard to retrospective memory and various executive functions (see McDaniel & Einstein, 2007). Studies of divergent validity in larger healthy and clinical populations are needed in order to establish the specificity of PROMACY (and MIST).

Contrary to our expectation, PROMACY did not correlate significantly with the NEPT and that null association was accompanied by a small effect size. This finding was surprising as both tasks demonstrate some evidence of discriminant validity in clinical populations (Harris et al., 2017) and ecological relevance (Sirois et al., 2016). From our vantage point, there are several possible reasons why these two PM measures did not correlate with one another in this study. First, it is possible that the current sample of non-clinical youth produced a very restricted range of scores that resulted in a ceiling effect and risk of Type II error. This was especially evident on the PROMACY Event-based trials for which low reliability coefficients were observed. Second, it is possible that these two tasks measure two different aspects of PM. The NEPT is a very naturalistic, habitual task that only includes event-based trials and has minimal retrospective memory demands. By way of comparison, PROMACY includes both time- and event-based trials that are interspersed with one another among eight different cue-intention pairings, which place considerable demands on strategic monitoring and retrospective memory (as shown with the correlational analyses, Table 4). In order to resolve this issue, future studies might: 1) examine the associations between these two clinical PM tasks with traditional experimental tests of PM, such as the classic McDaniel and Einstein paradigms (e.g., 2007), which would allow for a theory-driven analysis of component processes; and 2) measure the association between the NEPT and PROMACY in clinical samples with greater variability in PM scores.

As anticipated, a relationship with age was observed whereby older youth (≥ 15 years) scored higher on the PROMACY Summary Score than younger participants (< 15 years). This was a robust effect that is consistent with prior reports of improved PM performance associated with developmental progression in adolescents over children (Kliegel et al., 2013; Mahy et al., 2014; Voigt et al., 2014; Ward et al., 2005; Yang et al., 2011; Zimmerman & Meier, 2006). Furthermore, the positive age association observed here is consistent with findings from the MIST in adults (see Kamat et al., 2014; Raskin et al., 2010). Similarly, the absence of an association between PROMACY and gender and race/ethnicity aligns with adult MIST studies (e.g., Woods et al., 2008; Raskin et al., 2010; cf. Palermo et al., 2016). Of course, such findings do not preclude the possibility that gender and race/ethnicity may interact with other demographic or disease-related factors in influencing PROMACY performance.

With regard to component analyses, the most common error type observed in this small sample of healthy children and youth was omissions, followed by task substitution and loss of content. The majority of participants committed at least one error, with one-fourth of the sample committing four or more errors, suggesting a sizable subgroup had difficulty maintaining the intention, or alternately with execution. Nonetheless, participants performed well on post-task recognition of PROMACY items independent of PROMACY task performance. These distributions and error patterns align nicely with studies of the MIST conducted in healthy adults across the lifespan (e.g., Bezdicek et al., 2014; Kamat et al., 2014; Raskin et al., 2010; Woods et al., 2008).

This study is not without limitations. First, the study sample represents a cohort of healthy children and adolescents who were exposed to HIV in-utero, most of whom were also likely exposed to anti-HIV prophylactic medications postnatally. Thus, results may not generalize to other healthy children and adolescents without such exposures. Additionally, the sample was relatively homogeneous, being primarily Black, non-Hispanic, and impoverished, which may further limit generalizability to children and adolescents of other demographic backgrounds. The study sample did not include participants younger than 8 years of age; thus, future studies may wish to examine the potential usefulness of PROMACY in younger children. The sample also was relatively small and thus not of appropriate size for some psychometric analyses that will be important in future work (e.g., norms). Furthermore, scores obtained on neurocognitive measures for this cohort were generally within the Low Average to Average range of functioning, thus supporting their inclusion as the PROMACY validation sample. However, obtained PROMACY scores were relatively restricted in range, which may have limited statistical analyses. Range restrictions (and ceiling effects) are common among event-based PM tasks, especially those with low strategic demands. As noted above, it remains a challenge to develop a reliable PM measure with a sufficient number of intentions separated by a reasonable delay interval that is also brief enough to be implemented in a clinical setting with children and adolescents. Although our sample may represent children at risk for chronic health conditions well, replicating validation of PROMACY with a more representative demographic sample, that includes clinical samples, is needed. Including children and adolescents with cognitive, behavioral, and/or health conditions would likely address the observed score range restriction resulting in strengthened psychometric properties.

The ability to assess PM in children and adolescents is an important factor in fully understanding their developmental neurocognitive progression and daily functioning, but available clinical measures are non-existent. In this study, we present preliminary evidence of the internal consistency, convergent validity, and demographic correlates of PROMACY. When paired with two prior studies showing the discriminant and ecological validity of PROMACY (Harris et al., 2017; Sirois et al., 2016), the current findings are cautiously encouraging, particularly for the Summary Score. PROMACY is the first of its kind, easy to administer, comprehensive measure, that includes both Event-based and Time-based tasks of PM evaluated for use with children and adolescents. However, PROMACY needs to be further validated with a larger, more demographically diverse healthy sample and various clinical populations before being considered for clinical use. Future analyses will allow for validation of an alternate form and test-retest reliability, which will help inform longitudinal comparisons.

Figure 1:

Post-PROMACY Retrospective Recognition Task

*Note: PAR, Inc. holds the copyright to Prospective Memory Assessment for Children & Youth (PROMACY) and should not be used or reproduced without the written permission from PAR, Inc.