Online Information Search Performance and Search Strategies in a Health Problem-Solving Scenario

Abstract

Although access to Internet health information can be beneficial, solving complex health-related problems online is challenging for many individuals. In this study, we investigated the performance of a sample of 60 adults ages 18 to 85 years in using the Internet to resolve a relatively complex health information problem. The impact of age, Internet experience, and cognitive abilities on measures of search time, amount of search, and search accuracy was examined, and a model of Internet information seeking was developed to guide the characterization of participants’ search strategies. Internet experience was found to have no impact on performance measures. Older participants exhibited longer search times and lower amounts of search but similar search accuracy performance as their younger counterparts. Overall, greater search accuracy was related to an increased amount of search but not to increased search duration and was primarily attributable to higher cognitive abilities, such as processing speed, reasoning ability, and executive function. There was a tendency for those who were younger, had greater Internet experience, and had higher cognitive abilities to use a bottom-up (i.e., analytic) search strategy, although use of a top-down (i.e., browsing) strategy was not necessarily unsuccessful. Implications of the findings for future studies and design interventions are discussed.

INTRODUCTION

Consumers are increasingly becoming involved in the management of their health, having at their disposal an enormous number of health websites that contain information on diseases, treatment options, medications, and health maintenance (Fox & Duggan, 2013). Although web-based health information sources are intended to help people manage their health issues, the high volume of online health information can make finding and using such information challenging. More specifically, because health information on particular topics is often distributed across multiple websites, each with varying degrees of topical content and levels of detail (Bhavnani, 2005), the user may be required to locate, filter, organize, and integrate comprehensive information from multiple sources.

Online health information seeking can be especially taxing for individuals who are less computer literate or for those with low health literacy or lower cognitive abilities (Czaja, Sharit, Hernandez, Nair, & Loewenstein, 2010). There have been a number of studies which have examined how age, Internet experience, and cognitive abilities impact the use of online information. Generally, these studies have shown that older adults had more difficulty finding information and that people with greater Internet experience and higher cognitive abilities performed better on Internet information-seeking tasks (Czaja, Sharit, & Nair, 2008; Sharit, Hernandez, Nair, Kuhn, & Czaja, 2011; Trewin et al., 2012). Studies have also focused on Internet search tactics and site navigation strategies and have generally shown that the types of Internet search practices employed by the user significantly impact information retrieval (Hölscher & Strube, 2000; Navarro-Prieto, Scaife, & Rogers, 1999; Sharit et al., 2011; Stronge, Rogers, & Fisk, 2006; Thatcher, 2008). Many of these studies, however, either were not conducted within the health domain or were restricted to confined medical websites (Czaja et al., 2008; Fairweather, 2008; Kules & Xie, 2011), used simulated web search designs (Chin, Fu, & Kannampallil, 2009), or were limited to simple fact-finding tasks (Aula & Nordhausen, 2006).

In this study, we report on the search strategies that people adopt when using the Internet to solve a relatively complex health-related search problem. To enable diverse individuals to successfully perform these types of tasks, it is important to understand how variables, such as age, Internet experience, and cognitive abilities, impact search behaviors as well as measures of search performance.

According to information-foraging theory, people strive for efficient and effective performance in searching for information (Pirolli & Card, 1999). Typically, measures of search time, amount of search, and search accuracy are used to assess the efficiency and effectiveness of such task performance (Aula & Nordhausen, 2006).

One objective of this study was to examine the relationships among these measures and, in particular, to determine if longer durations and greater amounts of search are positively related to improved search accuracy as might be expected in decision-making scenarios that dictate more cautious examination of information (e.g., Wickens, Lee, Liu, & Becker, 2004). In complex web-based health information-seeking problem scenarios, however, individual variables, such as Internet skills, cognitive abilities, and age, may translate into more efficient as well as more accurate search performance.

Another study objective was to identify the search strategies used by participants and to determine if the type of strategy adopted was related to age, Internet experience, or cognitive ability. Our approach to identifying search strategies was twofold. First, we used the findings from related literature to define a fundamental set of strategies that people were expected to adopt and determined the extent to which our study participants adopted such strategies. Second, we developed a more comprehensive categorization of behavioral search strategies and explored the potential usefulness of this approach by using it to characterize the least successful and most successful searchers in the study sample.

Related Literature On Web-Based Behavioral Search Strategies

Prior to studies on web-based information search strategies, Marchionini (1997) distinguished between two broad types of strategies that people use to seek information stored in electronic databases: analytical strategies on one end of a continuum and browsing strategies at the other end. This conceptualization of search strategies was consistent with the bottom-up, top-down, and mixed Internet-based search strategies that Navarro-Prieto et al. (1999) identified.

A bottom-up strategy refers to an analytic style of searching in which specific keywords are extracted from the problem and used as search terms in a search engine; systematic examination of the search engine’s results are then pursued until the desired information is found. A top-down strategy corresponds to a more serendipitous browsing style of information search that relies primarily on using a device’s browser to follow links that seem promising or to explore meaningful categories of information found on an Internet site. The mixed strategy refers to use of both a top-down and a bottom-up strategy, by either alternating between these strategies or using both in parallel by maintaining searches in multiple windows. (From this point forward, the distinction between search strategies in terms of being either analytical or browsing will be referred to as bottom-up and top-down, respectively, as reflected in the current literature on Internet search.) Navarro-Prieto et al. (1999) found that less experienced users tended to initialize their searches with a top-down strategy and then transition to a bottom-up strategy toward the end of their searches. In contrast, they found that participants who were more experienced in web search used either a bottom-up strategy or an initial mixed strategy followed by a bottom-up strategy for fact-finding tasks but adopted a top-down strategy for an exploratory task. The idea of a mixed strategy was also alluded to by Hölscher and Strube (2000), who found that the web-based information-seeking behavior of experienced Internet users was characterized by frequent switching between following links and using search engines.

A further elaboration of bottom-up search strategies was provided by Drabenstott (2001), who classified the strategies that are used to formulate keywords in information search activities into the following six categories: (a) “shot in the dark,” in which a single word is used; (b) “Bingo”, which requires the user to enter a series of words; (c) “kitchen sink,” in which the searcher throws in “everything but the kitchen sink”; (d) “big bite,” which occurs when additional search terms are added to a previous search to narrow the results; (e) “citation pearl growing,” in which a searcher uses words and phrases from a successful search in a subsequent search; and (f) “help from your friends,” in which the searcher uses subject directories to aid in his or her query. These distinctions may also capture different attitudes among searchers regarding trading off between efficiency and thoroughness, as some approaches require expending less effort at formulation of search terms but at the expense of having large result lists returned for possible examination.

Eyesenbach and Köhler (2002) investigated how consumers searched the web to answer health-related questions, such as “If you want to travel to Australia do you need malaria prophylaxis?” Although it was more effective to use keyword searches characterized by combinations of terms (such as Australia malaria prophylaxis), the majority of keyword searches that were used consisted of a single word. Generally, one of the first results displayed by the search engine’s results page was used as a basis for iterative refinement of the keyword search process. No correlation between Internet experience and search time was found.

A study by Aula and Nordhausen (2006) involved 22 participants who performed five mostly fact-finding tasks (e.g., “How much blood does the human heart pump in one minute?”). They found that experienced users tended to employ a bottom-up strategy with longer queries (more keyword search terms) and iterated their queries when the initial search proved unsuccessful, reformulating queries based on both the previous search terms and the retrieved results. In contrast, less experienced users used more general keyword searches, and when these searches were iterated, the reformulations of queries were much less related to one another.

Differences in search strategies between younger and older adults were examined in a study by Stronge et al. (2006) in which participants were asked to perform tasks mostly related to finding locations, such as “Locate the website for General Motors.” Several strategies were identified, including a system tool strategy (for example, searching through the categories on the home page of a search engine, such as Yahoo); a keyword search strategy, which used the search engine’s keyword search capabilities; and a URL strategy, which involved typing a URL into or changing a URL within the address bar. The younger people were found to use significantly more keyword and URL strategies than the older participants, whereas the older adults used the system tool strategy to a significantly greater extent than their younger counterparts.

METHOD

Procedure and task description

Participants were recruited via word of mouth and flyers disseminated throughout the south Florida community. Inclusion criteria included fluency in English, 20/40 vision (corrected or uncorrected), and prior computer and Internet experience. Participants were also required to be non–cognitively impaired as measured by a score ≥26 on the Mini Mental Status Examination (Folstein, Folstein, & McHugh, 1975), a screen for cognitive status.

After providing informed consent for the study, which was approved by the university’s institutional review board, 60 participants completed a standard demographics questionnaire and a questionnaire concerning various aspects of their Internet use, including the degree to which they use the Internet (several times a day, about once a day, 3 to 5 days a week, 1 to 2 days a week, every few weeks, less often, and never). Participants also completed the short form of the Test of Functional Health Literacy in Adults (S-TOFHLA; Baker, Williams, Parker, Gazmararian, & Nurss, 1999); the scores of this instrument are categorized as inadequate, marginal, or adequate.

In addition, participants completed a cognitive battery containing tests of component cognitive abilities hypothesized to be important to Internet-based health information seeking (Sharit, Hernández, Czaja, & Pirolli, 2008). The battery measured six cognitive abilities: working memory (Alphabet Span; Craik, 1986), reasoning (Inference Test; Ekstrom, French, Harman, & Dermen, 1976), processing speed (Digit Symbol Substitution; Wechsler, 1981), verbal ability (Nelson-Denny Reading Comprehension Test; Brown, Fishco, & Hanna, 1993), executive function (Trail Making Test; Reitan, 1958), and visuospatial ability (Paper Folding Test; Ekstrom et al., 1976). The participants also completed a fact-finding Internet search task to ensure that they had basic Internet search skills. All participants were able to complete this task.

Participants were then given a booklet describing two health scenarios. Each was followed by a series of questions related to the scenario that required the participants to find and interpret information and in some cases make a decision. Participants were instructed to write their answers down in the spaces provided below each question. The first scenario was related to multiple sclerosis (MS; Table 1), and the second was related to diabetes. In this paper, we are restricting our analysis to the MS problem for the following reasons. First, we considered the MS problem scenario to be more closely aligned to the domain of problem solving given the greater degree of interdependency among its corresponding questions and the fact that this scenario contained both well-defined and more ill-defined aspects. The diabetes problem, in contrast, consisted of questions that were relatively independent of one another (e.g., “Does decaffeinated green tea have the same potential benefits as regular green tea towards managing type 2 diabetes?” and “What are some of the signs of an oncoming hypoglycemic attack?”). Second, whereas all participants completed the MS problem, a number of participants did not complete the diabetes problem, resulting in an incomplete data set for this problem.

TABLE 1.

Multiple Sclerosis Problem Scenario

Base Scenario	Eight years ago, your sister, now age 32 was diagnosed with Relapsing- Remitting Multiple Sclerosis (RRMS). To treat her form of Multiple Sclerosis (MS) she was prescribed a medication known as Avonex. She was functioning well until about a month ago when she started experiencing vision loss and muscle spasms in greater frequency. Upon visiting her physician, she learned that her Relapsing-Remitting Multiple Sclerosis had changed into Secondary Progressive Multiple Sclerosis (SPMS).
	A. What is Secondary Progressive Multiple Sclerosis (SPMS)? Please describe the disease and list four symptoms of SPMS.
	B. How does SPMS differ from the three other types of MS (Relapsing- Remitting MS, Primary-Progressive MS, and Progressive-Relapsing MS)? Please describe the differences in how each type of MS behaves.
Secondary Scenario 1	After a careful screening, the physician advised your sister to continue taking Avonex at 30 micrograms (mcg) once a week even though it did not stop your sister’s Remitting-Relapsing MS from transforming into Secondary Progressive MS.
	C. You are wondering if increasing her dosage of Avonex will be better for treating her SPMS. Is Avonex more beneficial in higher doses? Yes or No.
	D. You are also wondering if there is a more effective medication that is used to treat Secondary Progressive MS. If there is one, please name the medication.
	E. Your sister also has a pre-existing heart condition. Are you comfortable with the physician’s recommendation to continue treatment with Avonex? Why or why not?
	F. Your sister also takes the medication Norinyl, a form of birth control. Does Norinyl cause any harmful interactions with her current medication Avonex? Yes or No.
Secondary Scenario 2	You have also heard from a friend that alternative therapy, such as exercise and/or proper diet, can have a positive effect on your sister’s MS.
	G. Are there any specific types of exercises that are recommended for individuals with MS? Please list two.
	H. Are there any suggested dietary practices that can help minimize the symptoms of MS? Please list two dietary recommendations and describe how proper diet can be beneficial to MS.

Computing Search Time, Amount Of Search, And Search Accuracy

In this study, search time was defined as the total time a participant needed to complete the MS task. Amount of search was defined on the basis of two attributes: the number of distinct websites visited and the number of transitions between websites (where a single website can be visited multiple times). To assess the accuracy of search, three of the study investigators assigned an initial score for each individual question of the MS problem scenario. A final score for each question was subsequently arrived at through consensus.

A number of factors contributed to the designation of the search accuracy scores. Specifically, the well-defined, fact-finding questions were given less weight than the ill-defined questions, which required greater information integration. Also, questions comprising more subtasks were allocated more points than the questions that contained fewer subtasks. All questions left unanswered or answered incorrectly received a score of zero. Once scores were established for each question, the participants’ answer booklets were graded by two raters. Discordant scores were infrequent, but any that were encountered were discussed and mutually resolved. A search accuracy score, which was the aggregate of the scores for each of the individual questions, was then computed for each participant. To examine the relationships between search time, amount of search, and search accuracy, correlations using the Spearman’s rho statistic were computed.

Examining The Impact Of Individual Variables On Search Performance

Eight individual variables were analyzed in this study: age, Internet experience, and the six cognitive ability measures. The correlations between search accuracy and the continuous variables (i.e., age and cognitive abilities) were calculated using the Pearson correlation (r), whereas the correlation between search accuracy and the categorical variable (i.e., Internet experience) was calculated using Spearman’s rho (r_s). To investigate the potential impact of the eight individual variables on the task performance measures of search time, amount of search, and search accuracy, lower and higher groups of participants were established for each of these variables. For the measure of Internet experience, participants were divided into two categories of Internet use: higher (once a day or more) and lower (3 to 5 days a week or less). For age and each of the six cognitive ability measures, lower and higher groups were established based on median splits (for age, this procedure resulted in a younger as compared to an older group of participants). Means and standard deviations for the search performance measures were calculated for each of the higher and lower groups, and independent t tests were used to investigate differences between these groups for each of the performance measures.

Modeling Internet Information Seeking

As a basis for examining participants’ online search strategies, we developed a model of web-based information seeking that depicts the various and sometimes complex paths that individuals can navigate while searching for information on the Internet. It also indicates the cognitive abilities and relevant knowledge potentially needed for supporting search activities at various steps along these paths. The model (Figure 1) was derived based on the observation and analysis of a large and diverse sample of adults who performed web-based information-seeking tasks (Czaja et al, 2010; Sharit et al., 2011) and other information search models published in the literature (Dinet, Chevalier, & Tricot, 2012).

Figure 1.

Model of Internet information seeking. The dotted line (marked with an asterisk) connecting Position 5 to Position 7 applies when a link on a given website (e.g., About.com) is a “proxy search term,” that is, when clicking on the link results in a website-specific search for the word/phrase used in the link.

This model extends an earlier version (Sharit et al., 2008, reviewed in Dinet et al., 2012) that was restricted to search engine–based information seeking. The present model includes the option for users to identify potentially relevant URLs as a basis for initiating the information search process and for the use of devices, such as links, tabs, and menus, to engage in top-down (i.e., browsing-oriented) strategies. This extended model reflects the more recent data from the Pew Health Online 2013 report (Fox & Duggan, 2013), which indicates that 13% of users who seek health information from the Internet immediately pursue a health website. The model also shows that users can alternate between bottom-up strategies, which rely on formulating keyword searches in search engine search boxes, and top-down strategies.

As indicated in Figure 1, searches begin with an orientation to the task at hand and an identification of the important “facets” of the problem. Facets represent key concepts of the problem that can be expressed at various levels of abstraction, depending on the goals and cognitive abilities of the problem solver. For example, when identifying the facets of the problem “What is secondary progressive multiple sclerosis [SPMS]?” some searchers may choose multiple sclerosis or SPMS, whereas other searchers may choose secondary multiple sclerosis symptoms. Facet analysis has been described as “vital for selecting the Web search strategy with the greatest likelihood for retrieving relevant information” (Drabenstott, 2001, p. 19).

After identifying the important problem facets, the paths to information collection begin to diverge. In many instances, Internet users begin a search by entering problem facets into a search engine search box to generate Internet-wide results. They must then view the search engine’s list of results and determine the prospects of obtaining information that is consistent with their search objectives. If the results are perceived or evaluated to be unsatisfactory, searchers may need to reevaluate their query terms or rethink the problem prior to resuming their search activity. However, rather than use a search engine, searchers may choose to translate the problem facets directly into the URL for a relevant website. For example, if looking for information on SPMS, the user might go directly to www.nationalmssociety.org or WebMD.

Once searchers arrive at a website, they may use problem facets to identify relevant links or menus that will take them to the desired information, or they may enter the problem facets as search terms in the site-specific search box. For example, if using WebMD to find SPMS information, searchers can use links to information within the “Multiple Sclerosis Health Center” section of the website, or they could enter SPMS symptoms into the WebMD search box. In either case, searchers will need to evaluate the information obtained on the pages of the website to determine if it meets their search criteria. If it is not satisfactory, searchers may further explore the website in a variety of ways, decide to abandon their search on the site and begin again on a different site, or even choose to begin a new search using a search engine.

Characterizing Broad And Detailed Search Strategies

Using our proposed model of Internet information seeking (Figure 1) and findings and concepts from the related literature on search strategies, we classified participants’ search strategies in two ways: broad strategies and more detailed strategies. Building on the framework proposed by Navarro-Prieto et al. (1999), we anticipated that the broad strategies would fall along a continuum, from a bottom-up strategy on one end to a top-down strategy on the other end, with mixed strategies incorporating elements of both these extremes.

The possibility exists, however, for more detailed classifications of participants’ search strategies to be made, based on additional concepts from the literature (e.g., Thatcher, 2008) as well as on insights derived from our model of information search. To this end, we developed a detailed classification scheme based on the elements shown in Table 2, although for any broadly defined strategy, only a subset of the elements might apply.

TABLE 2.

Attributes of Search for Summarizing Search Strategies

Attribute of Search Strategies	Elements of Attribute
Known address strategy (KAS)	The user goes directly to a known address (i.e., URL) where they expect to find the needed information
Strategy for extraction of facets from the problem scenario (FS)	Use of specific terms or phrases in the problem statement as the basis for keyword search terms
	Use of keyword search terms that are an abstraction of or appear unrelated to the problem statement
	Use of a combination of literal and abstracted terms
Keyword terms strategy (KTS)	Drabenstot (2001) strategy: shot in the dark, Bingo, kitchen sink, big bite, citation pearl growing
Reformulation of keyword search terms (RKS)	Systematic reformulation, moving toward more specific reformulations
	Nonsystematic reformulations
Results lists search strategy (RLS)	Always starting from the top of the list
	Ignoring top-down order and selecting sources based on examination of the results
	Parallel hub-and-spoke strategy: instead of returning to the search engine’s results pages, several browser windows based on these results are opened in parallel and examined sequentially
Credibility assessment of websites (CA)	Tendency to use dot-gov and dot-edu sites
	Tendency to use dot-com and noncredible sites
	Specifically checking the credibility of the site
Uses search box within a website (WSB)	Top-down versus nonsequential examination of results
	Literal versus abstraction of terms
Navigation strategy and degree of orientation (NS)	Reliance on browser (back button)
	Reliance on links
	Signs of loss of orientation versus stable orientation

Identifying Participants’ Search Strategies

Guided by the model presented in Figure 1 and the search attributes specified in Table 2, we identified Internet search strategies by documenting the step-by-step search activities participants employed, such as typing a URL into the address bar, performing a keyword search, and selecting a search result. We captured the on-screen activity and keyboard/mouse input of each of the 60 study participants using Morae Recorder (Version 3.2.1, TechSmith Corporation). These data were then imported into Morae Manager (Version 3.3.2) for further analysis. Morae Manager lets the user insert markers at points of interest in the video. In addition, the user can attach notes to the marker for a particular event to describe the event in more detail. The Morae application thus provides both a means for visual analysis of marked recordings and a searchable, exportable list of markers. Figure 2 depicts a screen capture from a video of a participant’s search task performance, including inserted markers; Table 3 shows the exported data generated from the inserted markers and their corresponding notes.

Figure 2.

An example of a clipping of video analysis using Morae Manager. Each diamond in the timeline (outlined with a box at the bottom of the screen) represents a marker inserted for a specific action. See Table 3 for the data exported from this clipping.

TABLE 3.

Corresponding Data Exported From the Video Clipping Shown in Figure 2

Elapsed Time	Recording	Event	Notes
20:23.1	38018_TC	Marker	google search box; ‘avonex and heart’
20:28.1	38018_TC	Marker	1st result; www.thisisms.com/ftopic...
20:31.9	38018_TC	Marker	back to google
20:42.5	38018_TC	Marker	3rd result; www.iddb.org/drugs/avonex/
20:57.3	38018_TC	Marker	used control+F to find ‘heart’ on page
21:40.7	38018_TC	Marker	back button to google results
21:45.2	38018_TC	Marker	google search box; ‘copaxone and heart’
21:58.4	38018_TC	Marker	scrolls to select result; www.copaxone.com/aboutms/novantrone.aspx
22:13.5	38018_TC	Marker	back to google results
22:27.5	38018_TC	Marker	chooses 3rd result down from previous selection; www.drugs.com/copaxone.html

Note. The “Elapsed Time” column indicates the time in the video at which the action occurred, the “Recording” column indicates the participant to whom the video belongs, the “Event” column indicates that a marker was inserted by the reviewer, and the “Notes” column indicates notes that were added by the reviewer at the time of marker insertion.

Four members of the research team were involved in the analysis of the videos; during this process, the analysts were blinded to the participants’ ages and search accuracy scores. Guidelines for inserting markers and documenting participants’ actions in the notes were agreed upon through consensus, and when issues arose related to defining a participant’s actions, agreement was achieved through group discussion.

Using the Morae data, two of the research team members independently classified each of the 60 participants into one of five search strategy categories: strongly bottom-up, moderately bottom-up, mixed (part bottom-up, part top-down), moderately top-down, and strongly top-down. We chose to distinguish between a moderate and strong emphasis within the bottom-up and top-down search strategy categories as not all participants who were inclined toward one of the ends on this search strategy continuum necessarily conformed to purely bottom-up or top-down strategies. These distinctions, which provided a more realistic qualitative appraisal of participants’ search strategies, were not unexpected for the type of problem scenario investigated in this study, as some searchers may transition to a more flexible search strategy tendency (whether bottom-up or top-down) when facing increased information search complexity.

The initial concordance between the two raters was perfectly accurate in 22 of the cases. In 33 of the cases, there was a discrepancy that spanned one category (e.g., one rater classified a participant as strongly bottom-up whereas the other rater classified this participant as moderately bottom-up); in the remaining five cases, the discrepancy spanned two categories. Through discussion among the raters, consensus in all discrepant classifications was achieved.

Chi-square tests (2 × 5 categories) were performed for each of the eight individual variables to determine if belonging to a lower or higher group (two categories) for any given individual variable was independent of search category (five categories). However, because the expected number of observations in at least one of the cells in each of these eight tests was less than five, the data from the two bottom-up and two top-down categories were pooled, resulting in three search categories. Combining the data in this manner resulted in the three search categories that are most often used in the literature.

Finally, more detailed characterizations of search strategies were summarized for the five least successful and the five most successful performers in terms of search accuracy. We chose this subsample of participants as we expected the more detailed descriptions to be most sensitive to searchers with highly contrasting performance. In addition to the broader search strategy classification, these 10 participants were characterized according to various key attributes of search strategies (Table 2). The characterizations of these participants also included their search accuracy scores and their classification into either the higher or lower group for each of the eight individual variables. The two raters who had previously classified each of the 60 participants into one of the five broad search strategy categories collaborated in using the data from the Morae application to derive the more detailed attributes of search strategies.

RESULTS

A total of 60 adults (28 males and 32 females) ranging in age from 18 to 85 years (M = 49.6, SD = 19.5) participated in the study. All participants were found to have adequate health literacy as measured by the S-TOFHLA, with scores on this measure ranging from 28 to 36 (M = 34.9, SD = 1.5). As a result of the relative lack of variability in these scores, this measure was not included in any subsequent analyses.

The majority of participants reported using the Internet daily; almost half of the participants (48.3%) indicated that they used the Internet several times a day, whereas 23.3% of the participants indicated that they used it approximately once a day. The remaining participants reported using the Internet 3 to 5 days per week (13.3%), 1 to 2 days per week (6.7%), every few weeks (5.0%), or less often (3.3%).

There was a negative, though not statistically significant, relationship between search time and search accuracy, implying a tendency for better search accuracy to be associated with shorter times on task. In contrast, significant positive relationships were found between the number of distinct websites visited and search accuracy, r_s(60) = .27, p < .05, and the number of transitions between websites and search accuracy, r_s(60) = .26, p < .05.

Significant correlations were found between search accuracy and the following five cognitive abilities: reasoning ability (Inference Test), r(60) = .43, p = .001; verbal ability (Nelson-Denny Reading Comprehension), r(60) = .39, p = .002; visuospatial ability (Paper Folding Test), r(60) = −.28, p = .028; processing speed (Digit Symbol Substitution Test), r(60) = .28, p = .030; and executive function (Trails B), r(60) = −.28, p = .029. Working memory (Alphabet Span), r(60) = .21, p = .100; age, r(60) = − .01, p = .950; and Internet experience, r_s(60) = .20, p = .135, were not correlated with search accuracy.

Table 4 presents the means and standard deviations of search time, amount of search, and search accuracy for the entire sample as well as for participants in the lower and higher groups for each of the eight individual variables. Also presented are the results of the independent t tests used to investigate differences between the lower and higher groups of participants on each of the search performance measures. As indicated in Table 4, the participants in the older group did not significantly differ from their younger counterparts in search accuracy. Although the older participants took significantly more time to complete the problem, they demonstrated significantly greater efficiency in their searches than did the participants in the younger group as measured by the number of transitions between websites.

TABLE 4.

Search Time, Amount of Search, and Search Accuracy Scores as a Function of Age, Internet Experience, and Cognitive Ability

Individual Variable	Search Performance Measures
	Search Time		Amount of Search				Search Accuracy Score
			Number of Distinct Websites		Number of Transitions Between Websites
	M (SD)	p Value	M (SD)	p Value	M (SD)	p Value	M (SD)	p Value
Overall (N = 60)	57.5 (18.9)		10.7 (5.9)		28.8 (18.8)		18.8 (5.1)
Age
Younger (n = 31)	50.9 (17.7)	.005**	12.1 (6.6)	ns	33.8 (21.7)	.034*	18.9 (5.6)	ns
Older (n = 29)	64.5 (17.9)		9.2 (4.7)		23.5 (13.7)		19.1 (4.6)
Internet experience
Lower (n = 17)	63.8 (21.2)	ns	9.0 (5.9)	ns	23.7 (17.0)	ns	17.1 (5.0)	ns
Higher (n = 43)	55.0 (17.6)		11.4 (5.8)		30.9 (19.3)		19.5 (5.0)
Reasoning ability
Lower (n = 31)	60.9 (21.4)	ns	8.9 (5.2)	.010**	22.5 (15.6)	.006**	17.3 (5.4)	.018*
Higher (n = 29)	53.8 (15.4)		12.7 (6.0)		35.6 (19.8)		20.4 (4.3)
Verbal ability
Lower (n = 31)	63.0 (20.6)	.016*	9.0 (5.5)	.015*	23.8 (16.6)	.029*	17.6 (5.3)	ns
Higher (n = 29)	51.5 (15.1)		12.6 (5.8)		34.2 (19.8)		20.1 (4.6)
Visuospatial ability
Lower (n = 30)	62.7 (19.5)	.030*	8.5 (5.1)	.003**	21.9 (14.5)	.004**	17.5 (5.6)	ns
Higher (n = 30)	52.2 (17.1)		13.0 (5.9)		35.7 (20.2)		20.1 (4.6)
Executive function
Lower (n = 30)	61.0 (21.6)	ns	8.5 (4.3)	.002**	22.9 (14.5)	.014*	17.1 (5.1)	.008**
Higher (n = 30)	54.0 (15.5)		13.0 (6.5)		34.7 (20.9)		20.5 (4.5)
Processing speed
Lower (n = 30)	63.2 (19.2)	.018*	8.0 (4.5)	.000**	21.4 (14.7)	.002**	17.0 (5.1)	.005**
Higher (n = 30)	51.7 (17.2)		13.4 (6.0)		36.2 (19.8)		20.6 (4.5)
Working memory
Lower (n = 31)	60.8 (21.8)	ns	9.1 (6.2)	.022*	23.9 (19.8)	.035*	18.0 (6.0)	ns
Higher (n = 29)	53.9 (14.9)		12.5 (5.1)		34.1 (16.5)		19.7 (3.8)

Note. Significance (p) values are given for the two-sided, independent t tests used to compare the lower and higher groups with respect to the search performance measures.

^*p < .05.

^**p < .01.

Whereas the distinction between being in the lower- or higher-Internet-experience group was not found to have any significant impact on any of the search performance measures, the categorization of participants as lower or higher in cognitive ability was found to be influential. In particular, processing speed was significantly related to all the search performance measures: Participants in the higher-processing-speed group spent less time, visited more distinct web-sites, transitioned more often between sites, and achieved higher search accuracy scores than did their counterparts in the lower-processing-speed group (Table 4). Similar patterns were demonstrated for the other cognitive ability measures. Specifically, participants in the higher-executive-function and higher-reasoning groups exhibited significantly greater amounts of search and achieved higher search accuracy scores than those in the corresponding lower groups; participants in the higher-visuospatial and higher-verbalability groups exhibited significantly shorter search times and greater amounts of search than their counterparts in the lower groups; and participants in the higher-working-memory group exhibited significantly greater amounts of search than those in the lower-working-memory group.

The number of participants classified into each of the five search strategy categories based on whether they were in the lower or higher group for age, Internet experience, or cognitive ability are presented in Figures 3 and 4. These figures show a clear pattern of association between search strategy classification and the grouping of participants into lower and higher groups for each of the eight individual variables. Specifically, younger participants, participants with higher Internet experience, and participants with higher cognitive abilities were more apt to use either strongly or moderately bottom-up strategies.

Figure 3.

Number of participants adopting broadly categorized search strategies as a function of higher or lower classification on age and Internet experience.

Figure 4.

Number of participants adopting broadly categorized search strategies as a function of higher or lower classification on cognitive ability.

The patterns of association observed between search strategy classification (top-down, bottom-up, and mixed) and the classification of participants into lower and higher groups on the eight individual variables depicted in Figures 3 and 4 were significant in the case of age, χ²(2, N = 60) = 6.32, p = .043; Internet experience, χ²(2, N = 60) = 8.70, p = .013; visuospatial ability, χ²(2, N = 60) = 15.46, p < .001; executive function, χ²(2, N = 60) = 6.71, p = .035; and processing speed, χ²(2, N = 60) = 13.71, p = .001. Generally, the younger participants and the participants in the higher-Internet-experience and higher-cognitive-ability groups demonstrated a much stronger tendency to adopt bottom-up search strategies.

Finally, Table 5 summarizes the results of the more detailed classification of search strategies from the sample of 10 participants selected for this analysis, along with their corresponding broader search strategy categories, search accuracy scores, and classification into either the higher or lower group for each of the eight individual variables. Inspection of Table 5 indicates that regardless of performance these participants tended not to rely on a known web address, and there was about an equal tendency for the least and most successful performers to use either a systematic or a nonsystematic approach to reformulating keyword searches. These participants also did not appear to be differentiated by their keyword terms strategy, with the majority tending to use the Bingo strategy (where the user enters a series of words).

TABLE 5.

Summary of Comprehensive Search Strategies for the Five Least Successful and the Five Most Successful Performers on Search Accuracy

Group	Age	Internet Experience	Reasoning Ability	Verbal Ability	Visuospatial Ability	Executive Function	Processing Speed	Working Memory	Search Accuracy Score	Search Strategy Category	Attributes of Search Strategies
											KAS	FS	KTS	RKS	RLS	NS
Five most successful performers	Younger	Higher	Higher	Higher	Higher	Higher	Higher	Lower	30	Moderately bottom-up	No	Literal	Bingo	Sys.	Nonseq.	Stable
	Younger	Higher	Lower	Lower	Higher	Higher	Higher	Lower	27	Strongly top- down	Yes	Literal	Bingo	N/A	N/A	Stable
	Younger	Higher	Higher	Higher	Higher	Higher	Higher	Higher	27	Strongly bottom-up	No	Combo	Kitchen/ Bingo	Sys.	Non-seq.	Stable
	Older	Higher	Lower	Lower	Lower	Lower	Higher	Lower	25	Mixed	No	Combo	Bingo	Nonsys.	Non-seq.	Stable
	Older	Higher	Higher	Higher	Higher	Higher	Higher	Lower	25	Mixed	No	Literal	Bingo/ shot	Nonsys.	Non-seq.	Stable
Five least successful performers	Older	Lower	Higher	Higher	Lower	Higher	Lower	Lower	10	Moderately top-down	No	Literal	Shot	Sys.	Nonseq.	Stable
	Younger	Higher	Lower	Lower	Lower	Lower	Higher	Lower	10	Moderately top-down	No	Literal	Bingo	Nonsys.	Seq.	Stable
	Younger	Higher	Higher	Higher	Lower	Lower	Lower	Lower	10	Strongly bottom-up	No	Literal	Kitchen	Sys.	Non-seq.	Loss
	Younger	Higher	Lower	Lower	Lower	Lower	Lower	Higher	8	Strongly top- down	No	Literal	Bingo	Nonsys.	Non-seq.	Loss
	Older	Lower	Lower	Lower	Lower	Lower	Lower	Lower		Moderately top-down	No	Literal	Bingo	Nonsys.	Non-seq.	Loss

Note. KAS = known address strategy; FS = strategy for extraction of facets from the problem; KTS = keyword terms strategy; RKS = reformulation of keyword search terms; RLS = results lists search strategy; NS = navigation strategy; Sys. = systematic; Nonsys. = nonsystematic; Seq. = sequential; Nonseq. = nonsequential. The N/As seen in row 2 under columns RKS and RLS indicate that the participant did not reformulate keywords or follow a strategy in searching results lists because this participant used only two keyword searches during search.

However, in extracting facets (in the form of keyword search terms) from the problem statements, two of the five most successful performers used some degree of abstraction in formulating their keyword terms, whereas all five of the least successful performers used the problem statements literally as a basis for their keyword search terms. Also, the navigational strategies of the least successful performers were more indicative of loss of orientation.

DISCUSSION

In this study, we investigated the search time, amount of search, search accuracy, and information-seeking search strategies, as well as the impact of age, Internet experience, and cognitive abilities on search strategies and performance measures, in a group of adults who searched the web to resolve a health information problem. Overall, the tendency to trade off speed for amount and accuracy of search was not observed as might be expected when the problem scenario calls for relatively cautious examination of information and when time constraints are largely removed (e.g., Wickens et al., 2004). Instead, the findings indicated that the more successful searchers visited more distinct websites and made more transitions between websites but did so without having to invest more time in their information search activities. This economy in search time, despite the greater investment in work effort, appeared to be due mostly to better cognitive abilities rather than greater Internet experience, underscoring the potentially critical role that cognitive abilities can play in complex Internet health information seeking.

Not surprisingly, the older participants spent significantly longer time performing their searches as compared to their younger participants, consistent with findings that older adults typically take longer to perform most tasks (Charness, 2009). However, even though the amount of search that these participants undertook with regard to the number of transitions between websites was significantly less than that of the younger participants, this behavior did not translate into significantly worse search accuracy; in fact, their search accuracy scores were slightly higher.

In contrast to age and Internet experience, cognitive abilities had the strongest and most consistent impact on the search performance measures. In a previous study of web-based health information seeking (Czaja et al., 2010), reasoning was found to be the dominant cognitive ability in predicting search accuracy. However, in that study, other search performance measures were not examined, the range of participants’ ages was more restricted, and the health information search tasks that were used were not considered to be complex. In this study, processing speed, reasoning, verbal ability, executive function, and visuospatial ability were all significant factors in governing search task performance, with processing speed emerging as the only cognitive ability that was significantly related to all search performance measures.

The central importance of processing speed, especially for complex cognitive task performance, is understandable for reasons that Salthouse (1996) has termed the “limited time mechanism” and the “simultaneity mechanism.” When processing speed is slow—that is, when there is a decrease in the speed with which many information-processing operations can be executed—performance may become degraded because (a) some aspects of performance may not be able to be successfully executed (limited time) and (b) the products of earlier information processing may no longer be available to support later processing of information (simultaneity). For example, when engaged in complex web-based information seeking, slower processing speed may compromise the perceptual processing operation involved in attaching meaning to perceived visual information, which occurs through retrieval of knowledge stored in the searcher’s long-term memory (Wickens et al., 2004). This type of disturbance in perceptual processing could lead to incomplete knowledge concerning the task being performed. Slower processing speed should also make it less likely that the results of earlier information-processing operations would be available for simultaneous processing with search information attained at later points in time, adding to further disruption of task performance.

Although visuospatial ability was not expected to impact search performance to the same degree as some of the other cognitive ability measures, the relatively strong results for this ability are not that surprising. Pictorial visualizations, which are pervasive on the Internet and whose understanding visuospatial ability purportedly supports, can provide the opportunity for deeper processing of website content (Oestermeier & Hesse, 2000). Also, and consistent with cognitive theories of multimedia learning (e.g., Moreno, 2006), visuospatial ability is critical for the organization and integration of information in support of task performance.

What was surprising, however, was the absence of a significant impact of working memory ability on search accuracy. One possible explanation is that the nature of this search task did not have high working memory demands, as participants typically documented relevant information immediately as they located it. Still, participants with higher working memory visited more websites and transitioned more between websites than those with lower working memory, which suggests that for these types of search problems, working memory may have indirectly impacted search accuracy through enabling a greater amount of search to be undertaken.

The findings concerning characterizations of participants’ search strategies were largely consistent with the literature in that people with more Internet experience were more likely to engage in bottom-up strategies (Aula & Nordhausen, 2006; Navarro-Prieto et al., 1999) but were contrary to Trewin et al. (2012), who found that younger and older people were about equally disposed to using these strategies. The latter findings, however, were based on their older and younger participants being matched on “fluid” intelligence (which encompasses cognitive abilities, such as working memory, that are involved in new learning or problem solving); such matching was not implemented in our study. If the older group of participants in our study did in fact have lower fluid cognitive abilities than their younger counterparts, it might at least partly explain their greater tendency to use top-down strategies as compared to the younger participants.

Despite the findings indicating a strong relationship between a number of the cognitive abilities and the tendency to use bottom-up search strategies, as well as strong relationships between cognitive abilities and search accuracy, these findings should not be interpreted to imply that top-down strategies will necessarily be unsuccessful or should be avoided. As browsers, search engines, and the Internet become more powerful, the variety of paths that can be used to achieve Internet problem-solving success can enable a top-down strategy to be very efficient and successful, and indeed, we observed evidence of one such case.

Although one of the objectives of this study was to explore the use of a scheme for providing a much more elaborate characterization of users’ search strategies in this type of problem domain, it should be noted that such undertakings are clearly more arduous. In certain practical applications, however, pursuing such increased levels of detail in search processes may be imperative. For example, medical practitioners now rely on electronic resources, such as Google, for obtaining information relevant to their problem-solving and decision-making tasks (Davies, 2011; Duran-Nelson, Gladding, Beattie, & Nixon, 2013), often under time constraints, and thus understanding the nuances in search that may differentiate more successful from less successful searchers may constitute an important goal.

One limitation associated with this study was the lack of incorporation of a “think-aloud” procedure (Ericsson & Simon, 1993) in deriving search strategies, which potentially could have added a much richer set of data for characterizing search strategies. As the problem-solving task was considered to be relatively complex, we believed that the verbalizations of the participants’ thought processes could have possibly interfered with their task performance. This concern was heightened for the older participants, as a number of these people may have had lower cognitive capacities to begin with as compared to their younger counterparts. Another study limitation was the lack of data on participants’ domain knowledge, a factor that could impact the initiation of the Internet information-seeking process. Although none of the participants reported having MS, there was still the possibility that some participants may have had more knowledge about this disease than others.

In conclusion, despite the increasing power of search engines, health-related problem solving using the Internet remains a complex activity. This complexity derives in part from the cognitive demands that the underlying information-seeking processes entail as well as the characteristics of many health problem scenarios. In this regard, authors of future studies should consider how to more efficiently provide detailed characterizations of search strategies. Doing so, in turn, could provide insights into imparting more effective information search strategies to specialists and other users in various information search domains. More emphasis also needs to be given to the characteristics of search problems in these different domains to which the various search strategies might be differentially suited. Because we investigated a specific type of problem, one needs to be cautious about generalizing the results to distinctly different search problems, even within the health domain.

Consideration also needs to be given to new ways of supporting consumers of health information, especially older adults who are susceptible to normal age-related declines in cognitive abilities (Fisk, Rogers, Charness, Czaja, & Sharit, 2009). This study demonstrated that these abilities are potentially capable of facilitating more effective and efficient information search performance, perhaps in part through enabling the adoption of more effective search strategies. Cognitive abilities required for effective information seeking could possibly be supported through innovative browsing aids or through adaptive interfaces that suggest to the user how to approach information seeking based on features of the search problem. Designing such support, however, can be a challenging enterprise, as the incorporation of any new design feature needs to ensure that it in and of itself does not give rise to, directly or indirectly, new and excessive information-processing demands on the user.

Biographies

Joseph Sharit is a research professor in the Department of Industrial Engineering at the University of Miami. He is an investigator in the Center on Research and Education for Aging and Technology Enhancement (CREATE) in the university’s Center on Aging and a researcher with the Laboratory of E-learning and Multimedia Research at the Bruce W. Carter Veterans Administration. He received his PhD degree in industrial engineering from Purdue University in 1984.

Jessica Taha is a senior research associate at the Center on Aging at the University of Miami Miller School of Medicine. She earned a PhD in ergonomics at the University of Miami in 2012.

Ronald W. Berkowsky is a postdoctoral associate at the Center on Aging at the University of Miami Miller School of Medicine. He earned his PhD in medical sociology at the University of Alabama at Birmingham in 2014.

Halley Profita is a doctoral candidate in the Department of Computer Science at the University of Colorado at Boulder. She received a master of industrial design at Georgia Tech in 2011.

Sara J. Czaja, PhD, is a Leonard M. Miller Professor of Psychiatry and Behavioral Sciences at the University of Miami Miller School of Medicine and Industrial Engineering at the University of Miami. She is also the director of the Center for Research and Education for Aging and Technology Enhancement (CREATE) and the Scientific Director of the Center on Aging at the Miller School of Medicine. Her research interests include aging and cognition, e-health, caregiving, human–computer interaction, and functional assessment.