The Neuropsychological Aspects of Performance-based Internet Navigation Skills: A Brief Review of an Emerging Literature

Abstract

Objective:

Over the last 20 years, the internet has become a fundamental means by which many people with neurocognitive disorders manage their activities of daily living (ADL; e.g., shopping) and engage in health behaviors (e.g., appointment scheduling). The aim of this review is to summarize the emerging literature on the neuropsychology of performance-based tasks of internet navigation skills (INS) as measures of everyday functioning.

Method:

We performed a structured, qualitative review of the extant literature on INS using PRISMA guidelines.

Results:

Seventeen peer-reviewed studies met inclusion criteria and their results suggest that performance-based tests of INS: 1) discriminate healthy adults from some neuropsychological populations (e.g., HIV, multiple sclerosis, traumatic brain injury); 2) are associated with performance-based tests of everyday functioning capacity, domain-specific declines in manifest everyday functioning, and self-reported internet behavior, but not global manifest functional status; 3) correlate with standard clinical neuropsychological tests, particularly executive functions and episodic memory; 4) may relate to demographic factors, most notably age; and 5) have largely unknown psychometric properties (e.g., reliability).

Conclusions:

This review provided early support for the construct validity of performance-based tasks of INS as modern measures of everyday functioning in neuropsychological populations. Future work is needed to refine these tasks, establish their psychometrics, and evaluate their construct validity in diverse populations, as well as to develop effective remediation and compensatory strategies to improve internet functionality among persons with neurocognitive disorders.

The assessment of everyday functioning is a fundamental aspect of clinical neuropsychology. In clinic, referral questions commonly focus on the extent to which neuropsychological deficits, if present, affect a patient’s daily life. Such determinations arise in the context of diagnosing neurocognitive disorders (e.g., Sweet et al., 2015), which usually require that the observed neuropsychological deficits interfere with normal everyday functioning (e.g., American Psychiatric Association, 2013). Neuropsychologists are sometimes asked to determine whether neuropsychological deficits affect patients’ abilities to return to work, live independently, or fully engage in healthcare activities, such as adherence to treatment (e.g., Rabin et al., 2016). Likewise, neuropsychologists can help bolster the “so what?” of clinical neuroscience research by assessing the downstream effects of neuropsychological deficits on activities of daily living (ADL; e.g., household management, automobile driving), social functioning, health behaviors, and quality of life. Of course, there is an evidence-based rationale for why clinical neuropsychologists are commonly tasked with questions related to everyday functioning: Neuropsychological deficits are independently associated with a modest increase in risk of a range of different everyday functioning problems in many clinical populations (e.g., Kalechstein et al., 2013; Reger et al., 2014).

Yet in a recent survey of clinical neuropsychologists (Rabin et al., 2016), not a single test of everyday functioning appeared amongst the top 15 measures most commonly used in practice. This, despite the fact that nearly half of the clinical neuropsychologists surveyed were simultaneously concerned that their standard armamentarium of tests do not have adequate ecological validity (i.e., veridicality; Chaytor & Schmitter-Edgecombe, 2003). The most commonly used approach to assessing everyday functioning was a clinical interview (27.2%), followed by adaptive behavior scales and ADL questionnaires (26.2%). In a separate survey (Rabin, Burton, & Barr, 2007), only one-third of clinical neuropsychologists used ecologically-oriented tests (i.e., performance-based tests with naturalistic stimuli), mostly in the context of rehabilitation. The limited use of everyday functioning tasks among clinical neuropsychologists likely stems from a variety of factors, including minimal training in the assessment and remediation of everyday functioning (e.g., Hannay et al., 1998). In addition, there are myriad valid criticisms of the tools that neuropsychologists use to measure everyday functioning. Direct observation methods for assessing everyday functioning (e.g., smart homes, directly observed therapy) are resource-intensive and are not easily standardized or deployed in clinic (e.g., Dawadi et al., 2016). Self-report measures of manifest everyday functioning (i.e., what the patient actually does in daily life) are hampered by issues of poor insight, negative affect, and response bias (e.g., Bryant et al., 2018). Some performance-based tests of everyday functioning capacity (i.e., what the patient can do under ideal circumstances) can overcome the limitations of self-report measures, but are nevertheless criticized as: 1) being “glorified” neuropsychological tasks; 2) lacking adequate demographically-adjusted normative standards; and 3) using highly artificial and/or outdated materials and stimuli (e.g., Ardila, 2013; Marcotte et al., 2010). For example, some performance-based tests of financial management capacity use check writing tasks, which in the modern era are no longer as relevant to younger persons (e.g., Gerdes & Walton, 2005).

Thus, there is a need to develop, validate, and deploy modern tests of everyday functioning that will be relevant to the daily lives of our patients and useful to the next generation of clinical neuropsychologists. The internet will invariably play a major role in such efforts. The widespread availability and usage of high-speed internet has changed the way that people complete daily household, social, and health-related activities. Many of us now shop, socialize, bank, communicate, research goods and services, and coordinate our healthcare online (e.g., Ryan & Lewis, 2017). The internet is also increasingly being used for psychological and medical assessment and treatment delivery (e.g., Wawrzyniak et al., 2013). Accordingly, it is plausible that internet navigation skills (INS) represent an important, but presently overlooked aspect of everyday functioning for a variety of neuropsychological populations. Persons with neurocognitive disorders might use the internet to gather health information (e.g., symptoms, medication side effects), manage their healthcare (e.g., check laboratory results, schedule an appointment, manage a prescription), access psychosocial resources (e.g., online support groups), and complete ADLs (e.g., shopping, financial management) (e.g., Dorner et al., 2014). Most of us have experienced the frustrations of forgetting site-specific user names and passwords and have struggled to navigate non-intuitive or temperamental websites; for example, websites that only work with specific browsers or devices, frequently crash, contain faulty links, have unclear instructions, or have very long loading lag times. These design issues can be challenging and frustrating even for individuals with above average cognitive abilities, making it easy to appreciate their potential to complicate the lives of persons with neurocognitive disorders. For example, it is plausible that sensorimotor deficits could affect the way one engages with computer peripherals (e.g., keyboard, mouse, monitor), whereas higher-order neurocognitive deficits (e.g., executive functions, episodic memory) could interfere with one’s capacity to engage in and problem-solve an internet-based transaction. Indeed, two early functional neuroimaging studies also suggest that INS (i.e., online information searches) are related to brain activation in the prefrontal and temporal cortices (e.g., Dong & Potenza, 2015; Small et al., 2009). Moreover, neuropsychologically vulnerable populations have high rates of self-reported problems independently navigating technology (e.g., Nygard et al., 2012), including the internet (e.g., Mayben & Giordano, 2007).

Deficits in INS might therefore be a major barrier to optimal quality of life in persons with neurocognitive disorders. Of course the fields of cognitive psychology (e.g., Atkinson & Shiffrin, 1960) and clinical neuropsychology have historically embraced technology in many ways, with computing advances inspiring cognitive theory in the 1970s (e.g., Atkinson & Shiffrin, 1968), the computerization of task stimuli and scoring in the 1990s (e.g., Heaton, 1993), the development of internet-based platforms for assessment (e.g., Grosch et al., 2011) and scoring (e.g., Daniel, 2012) in the 2000s, and the advancement of virtual reality paradigms in more recent years (e.g., Parsons, 2015). However, to date we know relatively little about the neuropsychology of INS specifically. INS are worthy of investigation because they can provide a performance-based measurement of a critical and ubiquitous facet of everyday functioning in a non-artificial, ecologically relevant manner. The aim of this structured review was to summarize the emerging literature on performance-based INS tasks in the context of clinical neuropsychology, including their psychometrics, sensitivity to central nervous system (CNS) disease, and their associations with neuropsychological abilities, everyday functioning outcomes, demographics, and mood.

Methods

We used PRISMA standards (Liberati et al., 2009) to guide this structured review of the INS literature. We systematically queried the PubMed, Web of Science, and Google Scholar search engines from January 1997 through February 2018 using variants of these key words and their domain components: internet, World Wide Web, cognition, neuropsychological, everyday functioning, and quality of life. Our inclusion criteria were that an article: 1) be published in English in a peer-reviewed journal; 2) report data on a performance-based INS measure in human participants; and 3) report data on at least one cognitive or everyday functioning measure in relation to a performance-based INS measure. Our on-line search yielded a total of 36 potentially eligible abstracts that were selected for full-text review. Articles were reviewed by two independent raters with disagreements vetted by the lead author. Fourteen (38.9%) studies were excluded because they did not include a performance-based measure of cognition or everyday functioning. Seven (19.4%) studies were excluded because they did not directly examine the relationship between an INS and a measure of cognition or everyday functioning. Finally, we reviewed the references and subsequent citations of all 15 remaining eligible articles, which yielded 2 additional eligible articles. Thus, 17 studies met our inclusion criteria.

In summarizing these 17 INS studies below, we made every effort to consider not only simple tallies of studies that showed an effect by way of traditional null hypothesis significance testing, but also the independence and magnitude of those effects (irrespective of its significance at the univariate level). To accomplish this objective, we extracted and/or calculated effect size data for each study in this review for which it was possible to do so. We considered both accuracy and speed measures from each study, but prioritized accuracy measures where possible to maximize across-study consistency. In some instances, we were able to contact the study authors for this information. For neuropsychological outcomes, we extracted the effect size from a summary composite, domain-based measures where possible. When this was not possible and multiple indices were available, we selected the effect size from what the authors judged to be the most reliable and valid single index based on the extant literature. Of course, this is a small, emerging literature with considerable heterogeneity in sampling, methods, and populations, so we approach the effect size interpretation cautiously and do not equate this descriptive method with that of a meta-analysis. Instead, our goal was to adhere to long-standing recommendations in neuropsychology (e.g., see Bezeau & Graves, 2001) to consider effect sizes in interpreting study findings. Thus, we calculated Hedges g or Pearson’s r, as dictated by the study design, and used standard conventions for small (g < .2; r < 1), medium (.2 ≥ g < .5; .1 ≥ r < .3), and large (g > .8; r > .5) effects (see Bezeau & Graves, 2001),

Results

Table 1 provides descriptive details (e.g., author, date, sample characteristics, measurement of INS, and primary findings across the correlates of interest) on the 17 studies that met our inclusion criteria. According to the NIH Quality Assessment Tool for Observational Cohort and Cross-Sectional Studies system, all 17 studies were rated as “fair”. The primary limitations of these studies were their use of convenience samples, cross-sectional designs that were not randomized or blinded, and the absence of power analyses. Below we review this literature in detail, beginning with the studies on internet search skills, most of which were conducted in healthy adults. We then focus on the studies that examined INS in clinical populations, which are organized by INS domain (i.e., shopping, finances, health tasks, and general navigation) and then by population. The next few sections then review the relationship of INS to everyday functioning, specific neuropsychological domains, and mood/affect. Finally we review the few studies that have examined the psychometrics of INS tasks.

Table 1.

Descriptive summary of the 17 studies examining neuropsychological aspects of Internet navigation skills (INS).

Authors	Sample(s)	Internet Navigation Skills Measure(s)	Primary Outcome	Neuropsychological Domain Associations									Everyday Functioning		Other Variables
			Sensitivity to clinical sample	Motor Skills	Proc Spd	Attn/WM	Learn	Mem	Lang	Ex Fx	Spat	Glob	PB	SR	Sex	Edu	Age	Mood
Naturalistic Online INS Tasks
Agree et al., 2015	Healthy adults (n = 223)	Health information search task
Austin, Kaye, & Hollingshead, 2017	Healthy adults (n = 42)	Naturalistic search monitoring task
Chevalier, Dommes, & Marqué, 2015	Healthy younger (n = 10) and older (n = 10) adults	Health information search task
Czaja et al., 2010	Healthy younger (n = 10) and older (n = 40) adults	Health information search task
Dommes, et al. 2011	Younger (n = 20) and older (n = 19)	Information search task
Goverover et al., 2010	Multiple sclerosis (n = 21) and healthy (n = 18)	Actual Reality: airline shopping task
Goverover et al., 2014	Multiple sclerosis (n = 18) and healthy (n = 16)	Actual Reality task awareness: airline and cookie shopping task
Goverover et al., 2015	Traumatic brain injury (n = 10) and healthy (n =10)	Actual Reality: cookie shopping task
Goverover et al., 2016	Multiple sclerosis (n = 41) and healthy (n = 32)	Actual Reality: airline and cookie shopping task
Goverover et al., 2017	Multiple sclerosis (n = 19) and healthy (n = 19)	Actual Reality: airline ticket shopping task
Sharit et al., 2008	Healthy older (n = 40) and younger (n = 10)	Health information search task
Sharit et al., 2015	Healthy adults (n = 60)	Health information search task
Experimenter-Designed INS Tasks
LaBerge & Scialfa, 2005	Healthy adults (n = 41)	Travel information search task
Haesner et al. 2015	Mild cognitive impairment (n = 25) and healthy (n = 25) adults	Basic navigation skills and communication task
Pak & Price, 2008	Younger (n = 50) and older (n = 50) adults	Travel information search task
Woods et al., 2016	HAND (n = 19), HIV+ (n = 27), HIV− (n = 21) adults	Health pharmacy and medical records tasks
Woods et al., 2017	HAND (n = 43), HIV+ (n = 50), HIV− (n = 42) adults	Shopping and banking tasks

Note: WM = Attention/Working Memory; Learn = Learning; Mem = Memory; Lang = Language; Ex Fx = Executive Functions; Spat = Visuospatial; Glob = Global; PB = Performance-based; SR = Self-report; Edu = Years of Education; INS = Internet Navigation Skills; HAND = HIV-associated neurocognitive disorder; HIV = Human immunodeficiency virus. Circles represent observed effect sizes, irrespective of results of null hypothesis significance testing. Squares represent the results of null hypothesis significance testing when effect sizes could not be generated.

= No to small effect size

= Small to medium effect size

= Medium to large effect size

= Large effect size

= Non-significant finding, effect size

= Significant relationship, effect size not available not available

Internet Search Skills and Cognition

To our knowledge, Laberge and Scialfa (2005) were the first to report an association between neuropsychological measures and a performance-based INS task. Forty-three healthy participants searched an experimenter-designed and controlled travel website for 12 factual items (e.g., “What are the current hours of operation for Canadian Olympic Park?”). Specific subject knowledge and internet familiarity were strong predictors of INS search task performance. Vocabulary was not associated with any INS score, but age, working memory, spatial ability, and information processing speed were all related to INS search time per trial (medium-to-large univariate effect sizes). Verbal working memory and processing speed were also related to the number of pages visited during the INS search task (generally small-to-medium univariate effect sizes). Thus, this early study provided initial support for the idea that individual differences in cognitive functions could contribute to INS task performance.

Overall, there were nine articles that examined the relationship between neuropsychological functions and internet information-seeking behavior using performance-based INS metrics (Agree et al., 2015; Austin et al., 2017; Chevalier et al., 2015; Czaja et al., 2010; Dommes et al., 2011; Sharit et al., 2008; 2015; Laberge & Scialfa, 2005; Pak & Price, 2008). Nearly half (44%) of the studies included health-related topics in the search paradigms (e.g., Sharit et al., 2008; 2015), such as vaccinations, Multiple Sclerosis (MS), obesity, and exercise. All nine of the internet search studies were conducted in healthy, nonclinical samples and most were primarily interested in the effects of age on internet search behaviors, using both group-based (e.g., Chevalier et al., 2015) and correlational designs (e.g., Leberge & Scialfa, 2005). In general, their findings are representative of the broader age-related technology literature (e.g., Czaja et al., 2001), showing that older adults are slower in completing internet-based search and tend to have greater difficulties with more complex INS search tasks. Of greater relevance to this review, eight of the nine articles found a significant, positive relationship between cognition and internet-based information-seeking behavior (e.g., Leberge & Scialfa, 2005). The anomalous study in this regard (Chevalier et al., 2015) nevertheless reporting a trend-level relationship between cognitive flexibility and INS search task accuracy. The neuropsychological findings from these nine search-based INS studies are considered below in an integrated fashion alongside the eight other INS studies included in this review.

INS Task Deficits in Neuropsychological Populations

Next, we describe in detail the eight studies that specifically studied INS in a neuropsychological condition, including human immunodeficiency virus (HIV; Woods et al., 2016; 2017), MS (Goverover et al., 2010; 2014; 2016; 2017), traumatic brain injury (TBI; Goverover et al., 2015), and mild cognitive impairment (MCI; Haesner et al., 2015). The INS task domains assessed in these studies were shopping (k = 6), banking (k = 1), general skills (k = 1), and health-related activities (k = 2). (NB. in two of the eight studies [i.e., Woods et al., 2016; 2017], the authors reported data on two different INS tasks, therefore the k values add up to 10). All six of the studies that examined the effects of neuropsychological group status on INS performance accuracy found significant differences that were accompanied by large to very large effect sizes (displayed graphically in Table 1). By way of comparison, only three of the six studies observed significant group differences on time to INS task completion (Goverover et al., 2015; 2016; Haesner et al., 2015), with a fourth study showing a trend-level effect in a small sample (Goverover et al., 2010). Much smaller effect sizes were reported for speed across these studies (data not shown). Below we review the methods and group-level outcomes of all eight INS studies in neuropsychological populations.

Internet Shopping Skills

We identified six studies that have examined INS using online-shopping platforms (see Table 1), which included four studies in MS (i.e., Goverover et al., 2010; 2014; 2016; 2017), one in HIV disease (Woods et al., 2017), and in one moderate-to-severe TBI (Goverover et al., 2015).

Multiple Sclerosis.

To our knowledge, Goverover and colleagues (2010) were the first to describe INS deficits in a neuropsychological population. They administered an online shopping task to 21 persons with MS and 18 demographically comparable healthy adults. The INS task, which was cleverly labeled “Actual Reality”, required participants to access a live internet site to purchase a round-trip airline ticket, with specific travel parameters (e.g., lowest airfare, during the Summer months). INS task performance was scored for the presence and severity of 36 possible errors, time to completion, and “cognitive abilities,” which was the examiner’s rating of the efficiency with which participants completed the task. Despite comparable levels of computer and internet experience, the MS group demonstrated worse performance across all metrics of the INS airline ticket shopping task. Specifically, the MS group committed significantly more errors, required more cues, was less cognitively efficient, and was slightly slower, on this INS airline shopping task than the healthy group.

Goverover and colleagues largely replicated these findings in 2016 using an overlapping sample of 41 persons with MS and 32 healthy adults, 39 of whom comprised the sample in the study reported above. However, 34 of these participants were new and completed a parallel version of the INS airline ticket shopping task, which instead asked them to purchase and deliver cookies online to a teenager (Goverover et al., 2016). Findings in this combined sample that completed both the airline ticket and cookie shopping task revealed comparable group-level effect sizes for errors, cues, cognitive efficiency, and completion speed to those which were observed in the Goverover (2010) sample that completed only the airline ticket shopping task.

Goverover et al. (2014) examined the metacognitive aspects of INS in a related and overlapping study of 18 persons with MS and 16 sociodemographically similar healthy adults. Using a combined airline ticket and cookie shopping task, they reported that persons with MS and healthy adults generated similar predictions about how they would perform on the INS task and post-test ratings of their INS task performance. INS shopping action step errors were strongly and significantly related to task predictions (r = .45), but not post-test ratings of performance (r = .10).

Traumatic Brain Injury.

Goverover and DeLuca (2015) studied the Actual Reality INS cookie shopping task (described above) in 10 persons with moderate-to-severe TBI and 10 healthy adults who were comparable in demographics and computer experience. Despite the small samples sizes, there were significant differences between the study groups on this task that were accompanied by very large effect sizes. Specifically, the TBI sample committed significantly more errors, required more cues, was less cognitively efficient, and was slightly slower in completing the task.

HIV Disease.

Our research group examined the effects of HIV-associated Neurocognitive Disorders (HAND) on the Simulated Market Task (S-MarT; Woods et al., 2017). Study participants included 42 persons with HAND, 50 HIV+ persons without HAND, and 41 HIV− sociodemographically similar adults (Woods et al., 2017). In the S-MarT paradigm, participants were given 40 minutes to independently navigate a simulated, experimenter controlled online superstore to purchase and ship eight household items (e.g., a kitchen utensil, pain reliever) within a predetermined budget using a mock credit card. Participants also had to navigate several instant messages and advertising “pop-ups” that appeared during the task. S-MarT was scored for task failures (i.e., inability to successfully check out within the time limit), number of correct items purchased, specific errors (e.g., log-in, credit card entry), and completion time. Independent of sociodemographics, comorbidities, and computer use and anxiety, persons with HAND were more than 10 times more likely to fail to complete the S-MarT task as compared to their HIV+ and HIV− counterparts. Among participants who finished the task, the HAND group purchased a significantly lower number of correct household items, but did not differ from the other two groups in terms of error rates or time to completion (with small effect sizes for these latter metrics).

Internet Financial Management Skills

Only one study to-date has included a performance-based INS task of financial management (Woods et al., 2017), which was conducted in 35 persons with HAND, 49 HIV+ persons without HAND, and 40 sociodemographically similar HIV− adults. This study used the Web-based Evaluation of Banking Skills (WEBS; Woods et al., 2017), in which participants log in to a simulated, experimenter-controlled online bank and perform several common transactions (e.g., transferring funds, setting up an automatic bill pay, and reviewing recent transactions for errors). Similar to other INS tasks, WEBS was scored for overall accuracy (i.e., number of correct tasks completed), errors (e.g., log-in mistakes), and time to completion. No participants failed to complete the task within the 20-minute time limit. Independent of sociodemographic factors, comorbidities, and computer use variables, the HAND group demonstrated significantly lower accuracy as compared to the HIV+ group without HAND. There was no significant effect of HAND as compared to HIV− adults on accuracy, although the accuracy difference between the two groups was associated with a medium effect size in the expected direction. The study groups did not differ in error rates or completion time and the associated effect sizes were small.

Health-related Internet Navigation Skills

Our research group reported data on a pharmacy task and an electronic medical records task in 19 persons with HAND, 27 HIV+ persons without HAND, and 21 HIV-comparison participants who were broadly comparable in terms of sociodemographics (Woods et al., 2016). On the Test of Online Pharmacy Skills (TOPS), participants logged into a simulated, experimenter-controlled online pharmacy after completing a brief version of the Medication Management Test-Revised (MMT-R; see Scott et al., 2011). Participants were instructed to refill an existing prescription, request to fill a new prescription, activate a pick-up reminder on a cellular phone, and check for possible drug interactions for their new prescription. TOPS was scored for total items completed (i.e., accuracy), errors (e.g., log-in failures), and time to completion. The study groups did not differ in TOPS error rates or time to completion, and the accompanying effect sizes for these analyses were small. The HAND group demonstrated significant, large effect size differences in TOPS accuracy relative to the other two study groups; in fact, none of the HAND participants received a perfect accuracy score on the TOPS. In the full HIV+ sample, lower TOPS accuracy was associated with higher HIV RNA in plasma (r = −0.47).

In this same study sample, our group also examined the effects of HAND on the Test of Online Health Records Navigation (TOHRN; Woods et al., 2016). Participants were instructed to log in to a simulated, experimenter-controlled electronic healthcare management interface to access their message center and read and interpret a message regarding recent laboratory results, which required them to schedule an appointment with their provider online. TOHRN was scored for overall accuracy, errors (e.g., log-in errors, intrusions), and completion time. Akin to the TOPS findings reported above, less than 6% of the HAND group successfully completed this task and HAND was associated with large, significant effect sizes for overall accuracy. There were no between-group differences in error rates or time to completion (with small accompanying effect sizes) on the TOHRN. Lower TOHRN accuracy was associated with higher levels of HIV RNA in plasma (r = −.32).

General Internet Navigation Skills

Haesner and colleagues (2015) reported data on a general INS task in 25 older (age 60+) community-dwelling, German adults with Mini-Mental State Examination (MMSE) scores between 25 and 28 (which they labeled “MCI”) as compared to 25 older adults with MMSE scores >28. It was not clear if the study groups were matched on demographics or computer use, but the cohort reported high overall levels of technology interest and usage. The protocol required participants to log-in to an experimenter-designed website and perform two communication tasks (e.g., post a message to an online forum) and two technical assignments (e.g., change mail settings). Despite most of the sample requiring at least some assistance to perform these tasks, the low MMSE group made significantly more errors, required more assistance, and was slower to complete the INS task. The accuracy and speed differences were accompanied by medium-to-large effect sizes.

INS Tasks and Everyday Functioning

One critical consideration of the potential usefulness of the INS paradigms is the extent to which they map onto well-validated measures of functional capacity, manifest everyday functioning, and quality of life. This review identified six studies that have examined this important question (Agree et al., 2015; Goverover et al., 2010; 2015; 2016; Woods et al., 2016; 2017). The overall findings from these early studies are promising, although there is clearly much work left to do in this important area.

The first logical question is whether tests of INS, which are simulated capacity measures, relate to manifest difficulties using technology. Several studies matched their clinical and healthy comparison groups on technology use, but did not explicitly examine the relationship between INS and manifest technology use, problems, or familiarity (e.g., Goverover et al., 2010). Agree et al. (2015) found that better internet search performance was associated with more frequent daily internet use. Likewise, our research group reported that lower accuracy scores on INS pharmacy and medical records tasks were related to less frequent technology use, including daily internet use (average d = .95) among HIV+ persons (Woods et al., 2016).

Another fundamental question is how well INS relate to well-validated, performance-based tests of functional capacity, which are laboratory tasks that simulate ADLs such as medication management, financial transactions, and communication. There are two articles that examined this question across four different INS tasks in HIV disease (Woods et al., 2016; 2017). Our group found that INS accuracy (average r = .36) was reliably related to the performance on the University of California San Diego Performance-based Skills Assessment (UPSA-B; Mausbach et al., 2007) and the MMT-R. Importantly, the associations between INS accuracy on the shopping and banking tasks were related to the UPSA-B independent of relevant co-factors, such as neurocognitive impairment, depression, and HIV disease severity (Woods et al., 2017). Thus, these early data support the convergent ecological validity of those specific INS paradigms. A similar pattern of correlations emerges between INS and performance-based measures of health literacy, which assess the extent to which people understand, process, and use health-related information (see Sorensen et al., 2012). INS accuracy is related to the fundamental aspects of health literacy (see also Agree et al., 2015), most notably numeracy (average r = .47), as well as more advanced tasks involving comprehension and use of health-related information (average r = .45), including the Newest Vital Sign (Weiss et al., 2005). Thus, the early research returns suggest that INS tasks show medium-to-large associations with other performance-based measures of everyday functioning and health literacy.

The waters get murkier in determining whether INS relate to manifest everyday functioning, which describes what people actually do in their daily lives (vs. capacity), and quality of life. The data so far suggest that INS may be associated with self-report of domain-specific declines in ADL dependence. In 2017, we reported that INS shopping and banking accuracy had overall moderate correlations with self-reported declines in banking (r = .27) and shopping (r = .45) in daily life, and that these associations were independent of neurocognitive impairment, depression, and HIV disease severity (Woods et al., 2017). However, in this same study, INS were not significantly related to a global, composite measure of manifest daily functioning comprised of general self-reported ADL dependence, cognitive symptoms, clinician-rated ADL problems, and employment status. Similarly, null associations between INS accuracy and self- and proxy-ratings of general manifest daily functioning and quality of life were reported in MS (e.g., Goverover et al., 2010; 2014) and TBI (Goverover et al., 2015). However, it is worth noting that the samples in those latter studies were quite small (Ns < 20) and several of the observed effect sizes fell in the small-to-medium range (e.g., rs = .25 - .38), raising questions about possible Type II error. We are aware of only one positive finding for INS speed and self-rated manifest everyday functioning, which was reported in persons with moderate-to-severe TBI (r = .58; Goverover et al., 2015). Interestingly, post-hoc analyses of the data in the Woods et al. (2017) paper undertaken for this review showed that time to complete the INS shopping and banking tasks were significantly associated with global functional status in the HIV+ sample (ds = .46 and .48, respectively), independent of neurocognitive impairment, depression, and AIDS diagnoses (ps < .05). Taken together, these findings provide preliminary evidence in support of the relationship of some aspects of INS to some everyday functioning outcomes.

Neuropsychological Correlates of INS Tasks

Table 1 shows the effect sizes estimates of the associations between INS and neurocognitive domains from the 17 studies included in this review. All five studies that looked at the association between INS and a global index of cognition found significant, typically large associations whether they used a within-group correlational approach (e.g., Austin et al., 2017; Goverover et al., 2016) or a between-group design (Haesner et al., 2015; Woods et al., 2016; 2017). As shown in Table 1, there were 14 studies that reported at least one domain-specific analysis linking neuropsychological functions to INS tasks, 13 of which reported at least one significant domain-level relationship (92.9%). For the purposes of this review we classified the domain-level analyses across eight ability areas using standard neuropsychological conventions: motor skills (k = 3), information processing speed (k = 11), attention and working memory (k = 9), learning (k = 5), delayed memory (k = 6), language and verbal fluency (k = 9), executive functions (k = 10), and visuospatial abilities (k = 6). Forty of these 59 (67.8%) study-level domain associations displayed in Table 1 showed significant relationships with INS tasks. However, there was clear domain-related variability present in these findings. The associations between INS and episodic learning (5/5), delayed memory (6/6), and executive functions (9/10) were quite reliable and tended to fall in the medium-to-large effect size range. On the other end of the spectrum, the weakest and least consistent associations with INS were observed in the domain of language (3/9), with most studies showing small effect sizes. In a similar vein, attention/working memory did not relate to INS reliably (3/9), but four studies had at least medium effect sizes (see Table 1). With regard to psychomotor functioning, 2/3 studies linking INS to basic motor skills (e.g., grooved pegboard) and 8/11 measuring information processing speed (e.g., digit symbol tasks) found significant relationships, the majority of which were associated with small-to-medium effect sizes. Likewise, visuospatial skills (e.g., line orientation) demonstrated mostly reliable associations with INS across studies (4/6), with generally medium effect sizes.

INS Tasks and Mood/Affect

Depression and other emotional factors are important considerations in assessing everyday functioning in neuropsychological populations (e.g., Marcotte et al., 2010). Mood and anxiety disorders may also affect the way that people approach using the internet in the real world (e.g., apathy, motivational issues, self-efficacy). Thus far, there is no strong or reliable evidence that INS task performance is influenced by general mood or affective distress. Our group reported that INS shopping and banking were unrelated to a summary measure of current affective distress in a large, mixed sample of persons with and without HIV infection (Woods et al., 2017). Three out of four studies to date have reported null associations between depression specifically and INS task performance. In the setting of HIV disease and MS, INS task accuracy was not significantly related to depression as measured by self-report symptom questionnaires (e.g., Goverover et al., 2010) or clinical diagnoses (Woods et al., 2016; 2017). The exception so far has been a single study in which depression demonstrated a large, independent association with higher error rates and lower efficiency in INS shopping among persons with moderate-to-severe TBI (Goverover et al. 2015). No studies, however, have focused on persons with current, severe depression or anxiety, who may be at higher risk of INS deficits.

Although INS task performance may not be related to general anxiety (e.g., Goverover et al., 2015), the story may be somewhat different for specific measures of technology-based anxiety. Two studies have reported that INS task accuracy (but not speed) is related to both state and trait computer anxiety in mixed samples of persons with and without HIV disease (Woods et al., 2016; 2017). Such findings are in line with reports from the reading and math anxiety literatures, which suggest that specific anxieties may be differentiated in meaningful ways from general symptoms of anxiety (e.g., Carey et al., 2017).

Psychometrics of INS Tasks

At this juncture, very little is known about the psychometric properties of the INS paradigms, evaluation of which will be critical in moving this literature forward. We are unaware of any studies that specifically examined the psychometrics of INS tasks, including such factors as internal consistency, test-retest reliability, or measurement invariance. We found only one study on practice effects of INS tasks. Haesner et al. (2015) found significant, same-day practice effects in speed and accuracy on the same form of general INS task, the magnitude of which was medium-to-large and did not differ across older adults with and without low MMSE scores. The intercorrelations of measures derived from two INS paradigms (i.e., shopping and banking) was reported in 93 HIV+ and 40 seronegative adults (Woods et al., 2017). The concurrent validity of the two approaches was supported by the presence of broadly medium across-test correlations (rs from .33 to .45) on measures of accuracy and speed, but INS shopping and banking errors were not related to one another. Within test correlations showed that total INS completion time was not significantly related to accuracy (rs from .13 to −.18). For INS shopping, errors showed medium-to-large size within-test relationships to accuracy and time (rs from .27 to .53). On the INS banking task, errors were only related to accuracy in the HIV+ sample (r = −.36) and otherwise showed small and non-significant associations with accuracy and time (rs from −.02 to −.08).

Demographic factors, in particular age, may play a role in INS task accuracy. For example, five of the eight studies that examined the relationship between age and internet search tasks found that older age was associated with worse performance. In clinical samples, the relationship between age and INS is less clear as only one study has reported such data. Specifically, Goverover et al. (2016) noted that older age within an MS sample had a small and non-significant association with INS shopping accuracy and speed. With regards to education and INS task performance, two studies reported medium effect sizes whereby higher levels of educational attainment were associated with better INS accuracy in HIV (Woods et al., 2016) and MS (Goverover et al., 2016), but not in healthy adults (e.g., Agree et al., 2015). In terms of gender, the effects on INS task accuracy appear to be small and elusive, with four out of five studies reporting null associations (e.g., Woods et al., 2017). We are unaware of any studies that have examined effects of race/ethnicity or socioeconomic status (SES) on INS accuracy or speed.

Discussion and Conclusions

Over the last 20 years, the internet has become a fundamental means by which many people with neurocognitive disorders handle their activities of daily living (ADL; e.g., shopping) and engage in health behaviors (e.g., appointment scheduling). It is conceivable therefore that INS, which may rely heavily on neuropsychological abilities, play an important role in the functional independence and health outcomes of patients with CNS disorders. This review described an emerging literature of 17 studies that examined the neuropsychological aspects of INS in both healthy and clinical populations. Overall findings provided some support for the relevance of performance-based INS tasks as modern measures of everyday functioning for use in neuropsychological populations.

INS tasks appear to be quite sensitive to many different forms of CNS injury. This review revealed reliable, large group differences between healthy adults and persons with MS, TBI, MCI, and HAND on raw accuracy scores from a range of INS tasks, including shopping, banking, and health-related behaviors. Less reliable, but still notable CNS group differences were observed on measures of INS task completion speed, which were more pronounced in the TBI and MS groups than in HAND. Overall, these INS task group differences were not better explained by demographic factors, comorbid conditions (e.g., mood), or computer familiarity and anxiety. Of note, the severity of neurocognitive impairment in these studies was mild-to-moderate (e.g., no studies focused on persons with dementia), suggesting that even subtle cognitive deficits can interfere with INS. This review also showed significant, medium-sized correlations between INS and some disease biomarkers (e.g., Woods et al., 2016). A notable limitation of this literature thus far is the small sample sizes of the clinical and comparison groups (i.e., study cells ranging from 10 to 50), which may yield unreliable findings or Type II error (e.g., for the speeded INS tasks). The INS literature is also limited by the few neuropsychological populations that have been studied thus far. There are a host of different medical, neurological, and psychiatric disease groups for whom INS are relevant across the lifespan and worthy of study. There are also major gaps in the types of INS that have been studied in neuropsychological populations, including internet-based search, communication, social, or health insurance navigation, to name just a few. For example, all of the nine studies on internet search behaviors were conducted exclusively in healthy persons, so we know little about its sensitivity to CNS injury or associations with neurocognition and everyday functioning outcomes in neuropsychological samples.

Importantly, the INS tasks reviewed herein demonstrated some evidence of convergent ecological validity. That is, lower INS task accuracy scores were related to real-world outcomes. At the univariate level, INS tasks showed broadly medium correlations with: 1) well-validated, performance-based tests of everyday functioning capacity; 2) self-reported internet behavior; and 3) self-reported declines in domain-specific manifest everyday functioning. In many cases, these associations were independent of global neurocognitive impairment, depression, general and functional domain-specific internet use, and disease severity (e.g., Woods et al., 2017). Such findings suggest that the INS tasks provide incremental ecological validity, which may one day support their use in clinical neuropsychological practice. Future studies may wish to specifically evaluate the incremental ecological validity of INS against the specific cognitive domains (e.g., executive functions) with which INS are most closely related (see below). It was somewhat surprising that INS accuracy was not reliably associated with summary measures of manifest functional status, which raises the possibility that INS tasks have limited relevance to broad measures of functional dependence at this time. Yet this interpretation seems unlikely considering how reliably INS tasks were related to functional capacity and domain-specific functional problems (e.g., Woods et al., 2017). Another interpretation of these null findings is that INS tasks provide unique information about everyday functioning that is not captured by the older, global self-report measures of ADL functioning that were validated in the pre-internet era, do not include modern items related to technology, and may therefore be insensitive to some functional problems that occur in the internet era (Woods et al., 2017). Ideally, INS tasks will be validated against observation-based everyday functioning measures (e.g., Dawadi et al., 2016) and knowledgeable informants, especially given the challenges in measuring ADLs by way of self-report, which can be biased by limited insight and mood. In addition, future work may explore the extent to which performing ADLs on the internet exacerbate (or maybe even remedies in some instances?) the types of everyday functioning difficulties that neuropsychological populations demonstrate on analogue tasks (Ruse et al., 2014).

INS tasks demonstrated reliable and strong associations with global measures of neuropsychological functioning, but more variable relationships with domain-specific metrics. Medium-to-large correlations were evident between INS and measures of episodic memory (e.g., verbal and visual learning and memory) and executive functions (e.g., cognitive flexibility, novel problem-solving) across studies. These findings map onto prior work suggesting that internet search behavior is associated with activation in prefrontal and temporal regions (e.g., Dong & Potenza, 2015; Small et al., 2009). The association between episodic memory and INS tasks is interesting, considering that most INS paradigms allowed the participant to view the instructions at all times, thus minimizing the demand on that ability area. On the other hand, tests of language (e.g., verbal fluency and vocabulary) and attention/working memory were not reliably or strongly related to INS. Falling somewhere in the middle were tests of psychomotor speed and visuospatial abilities. This variability in significance and effect size magnitude may reflect the variability in INS measurement, neuropsychological construct measurement, the specific populations studied, and sample sizes. For example, processing speed was reliably and robustly related to INS shopping accuracy in MS (Goverover et al., 2010; 2016) and TBI (Goverover et al., 2015), but not in HIV (Woods et al., 2017), where such deficits have declined in prevalence and severity in the modern treatment era (e.g., Heaton et al., 2011). Moreover, the studies reviewed here varied in their emphasis on speed versus accuracy, with most taking a naturalistic approach in emphasizing the latter over the former. In designing our own INS tasks, we did not emphasize speed because it was our impression (based on descriptive pilot work) that most individuals accomplished internet-based ADLs at their own convenience and pace. However, future studies may wish to explore issues of speed/accuracy trade-offs in the context of INS.

Future investigators are advised to carefully consider using basic (e.g., Miyake, et al., 2000) and applied (e.g., Sutcliffe & Ennis, 1998) cognitive theory to elucidate the potential interplay between cognitive demands of the specific INS paradigm and the neuropsychological profile of the population under study. The early studies reviewed herein may provide the foundation for hypothesis-driven studies that are informed by cognitive psychological theory and aim to elucidate the neuropsychological architecture of INS in various populations (see Woods et al., 2009). For example, what specific aspects of executive functions (e.g., shifting, inhibition, updating) and memory (e.g., encoding, consolidation, retrieval) underlie the performance on INS tasks and/or mediate/moderate their associations with real-world outcomes? Are there cognitively meaningful classes of error types that are particularly important in predicting poor functional outcomes? Might individual differences in general and/or domain-specific semantic knowledge influence the relationship between cognitive domains and INS task accuracy and speed (e.g., Hong, 2006; Sanchiz et al., 2017)? How do speed/accuracy trade-offs influence task completion and critical error rates? Moreover, future studies are needed to determine the neural substrates of INS using multimodal approaches that might include structural and functional neuroimaging (e.g., Small et al., 2009), blood and cerebrospinal fluid biomarkers of CNS injury, and host and disease-specific genetic markers.

The psychometric properties of INS tasks are not known at this time, which is a major limitation of this literature and therefore a critical direction for future work. As field, we must establish the internal consistency, measurement invariance, test-retest reliability, practice effects, and factor structure of INS tasks in order to provide a stronger foundation upon which to evaluate their construct validity. Important questions remain whether INS are separable from and provide incremental ecological validity relative to general computer and technology skills. Moreover, standardization and norming of technology-based tasks is notoriously challenging, as these processes invariably lag behind rapid advances in speed, equipment, and software. For example, all of the INS tasks reviewed above use desktop computers, whereas many people currently access the internet on their mobile devices (e.g., Wilmer et al., 2017). Moreover, there are costs and benefits to using the open internet (e.g., naturalistic, but concerns over variable speed and content) versus experimenter-controlled (e.g., less naturalistic, but greater control over speed and content to allow for more mechanistic questions) web-based platforms for INS studies (see Table 1). Nevertheless, wide-spread use of INS tasks will depend on their standardization and subsequent investigation in nationally representative healthy samples to evaluate their psychometrics. Such efforts may include large-scale investigations of their associations with demographic factors, notably age and socioeconomic status (e.g., education), in order to determine what, if any, normative adjustments are needed for diagnostic purposes. It is not presently known which sociodemographic (e.g., age, education, sex, ethnicity, SES) and engagement (e.g., use frequency) factors are associated with the development of INS. The investigation of INS tasks may be especially relevant for under-served racial/ethnic populations, who are less likely to have regular access to high-speed internet and struggle using it independently (e.g., Maybin & Giordano, 2007). In fact, INS (e.g., search paradigms) may provide a unique, ecologically-relevant forum in which to study functional capacity across different cultures using a population’s native language(s) and familiar materials.

This review has several limitations worth considering. First and foremost, the studies reviewed herein were restricted to those that included both a performance-based measure of INS and a neurocognitive task. Therefore, we cannot directly speak to the findings from literatures that have used self-reported measures of INS, use, or anxiety in neuropsychological populations. Second, the studies included in this review were largely cross-sectional, observational designs thus making it difficult to determine whether the apparent deficits in INS represent a problem in skill acquisition or a decline from previous levels of functioning. Future studies would be wise to consider the implications of premorbid intellectual, neuropsychological, and INS functioning, which may mediate or moderate the association between current cognitive deficits and INS performance. Third, there was considerable heterogeneity in study design, measurement (of INS and cognition), analysis, and populations among the 17 studies reviewed. Although we relied on effect sizes where possible to complement our review of null hypothesis significance testing findings, we did not conduct a formal meta-analysis and therefore the aggregate effect size estimates should be interpreted with caution. At this early stage, we elected to be over-inclusive in capturing as many studies as we could that directly measured both of the constructs of interest, but the logical consequence that approach was heterogeneity in study methodologies and thus less precise aggregates of effect size estimates.

Finally, it is important to consider not only how INS tasks can be used to aid in diagnosis and prediction of potential problems with everyday functioning, but also how they might help us minimize and remediate cognitive decline (e.g., Tun & Lachman, 2010). For example, INS tasks may provide an ecologically valid outcome measure of everyday functioning for clinical trials of psychological (e.g., cognitive or behavioral) or medical (e.g., pharmacological or surgical) interventions designed to improve cognition. It is also important to identify, operationalize, and make readily available effective analogue work-around options for individuals with severe INS deficits, technology anxiety, and/or limited access. We may also use information gleaned from INS studies to improve the design and usefulness of websites for persons with neuropsychological disorders (e.g., Haesner et al., 2015), with an eye toward improving health engagement, outcomes, and quality of life. Clinical neuropsychologists should occupy an influential seat at the multidisciplinary table of applied and basic scientists that are leading the charge to create, validate, and deploy the next generation of INS tasks.