Using the Internet to Search for Cancer Clinical Trials: A Comparative Audit of Clinical Trial Search Tools

Abstract

Advancing the clinical trial research process to improve cancer treatment necessitates helping people with cancer identify and enroll in studies, and researchers are using the power of the Internet to facilitate this process. This study used a content analysis of online cancer clinical trials search tools to understand what people with cancer might encounter. The content analysis revealed that clinical trial search tools were easy to identify using a popular search engine, but their functionality and content varied greatly. Most required that users be fairly knowledgeable about their medical condition and sophisticated in their web navigation skills. The ability to search by a specific health condition or type of cancer was the most common search strategy. The more complex tools required that users input detailed information about their personal medical history and have knowledge of specific clinical trial terminology. Search tools, however, only occasionally advised users to consult their doctors regarding clinical trials decision-making. This, along with the complexity of the tools suggests that online search tools may not adequately facilitate the clinical trials recruitment process. Findings from this analysis can be used as a framework from which to systematically examine actual consumer experience with online clinical trials search tools.

INTRODUCTION

Both an extensive body of literature and current national strategies for combating cancer attest to the known value of clinical trials for improved prevention, early detection, and treatment [1, 2]. Yet, clinical researchers have historically faced difficulty recruiting the patients needed to complete trials in a scientifically sound and timely way. The many barriers to recruitment range from patients’ fears about receiving a placebo to a lack of awareness about cancer clinical trial opportunities [3]. One national survey conducted in 2001 reported by Harris Interactive and cited elsewhere [4, 5] found that 85 percent of cancer patients were either unaware of clinical trials or unsure about whether such trials were an option for them. Although the general public has positive attitudes about clinical trial research, they also have a generally limited understanding of the clinical trials process [6]. Recently, unpublished data gathered by NCI from a national sample confirmed that people were unaware of clinical trials, but those who were considering such trials were likely to use the Internet to find out about them.

The clinical trial search process serves as another barrier, even for those who are interested in clinical trials [7, 8, 9]. In order to identify potential trials, patients must comprehend the medical terminology related to cancer diagnosis and treatment. They must understand the clinical trial research process in general and learn the demands of a specific trial to determine their interest in and eligibility for that trial [9]. As a result, patients have been frustrated in their attempts to join a trial [10].

Clearly, a great many challenges prevent clinical trials recruiting. Some of them are information-based, and some relate to the search process itself. In considering the ways in which people may learn about clinical trials and gain access to them, the Internet appears to be an obvious medium given its importance as a consumer health information resource. Sixty-five percent of Internet users reported noticing health information on the Internet [11], while 79% of Internet users have actively searched online for information on at least one major health topic [12]. Further, cancer patients are both willing to use the Internet to search for clinical trials [13] and are actively doing so [14].

Websites devoted to cancer clinical trials are proliferating, but researchers are only beginning to examine the Internet’s potential for helping people locate clinical trials. Simon and Hegedus [15] examined the content of websites providing information about cancer clinical trials located using popular search engines. Many of the 66 sites they reviewed provided detailed information about cancer clinical trials, and almost half of the sites offered some kind of clinical trials search function. They concluded that clinical trial websites provided many benefits, including wide access to information about clinical trials and to clinical trial search tools, which might ultimately increase clinical trial recruitment. However, the amount and diversity of information, the complexity of the language used, and the lack of confidentiality and accreditation resources may pose barriers to finding a trial.

The findings related to language complexity were confirmed in a study [16] of the clinical trials search tools on the websites of NCI Comprehensive Cancer Centers. The researchers found that the search mechanisms on most sites allowed the user to search by type of cancer, but they often used medical terminology rather than lay language, such as melanoma instead of skin cancer. A readability analysis showed that most of the search results were written at an 11^th grade level, which is often too high for the general public given that about 90 million adults have literacy skills that test below the high school level [17].

Little other research has been conducted on the acceptance and use of Internet-based clinical trial search tools. Although one study [18] described one of the first online matching tools (University of Pennsylvania Cancer Center’s OncoLink site), they only gathered information from patients who were matched to clinical trials. No information was available on those who did not find a trial or even how many had attempted to use the matching tool.

The National Cancer Institute (NCI) was one of the first federal agencies to disseminate health information on the Internet. NCI’s website (www.cancer.gov) and its clinical trials search tool (www.cancer.gov/clinicaltrials/search) have been continually updated and refined based on multifaceted data collection strategies [14]. Given the ongoing challenges of recruiting for cancer clinical trials and the importance of the Internet as a consumer health information source, this article represents NCI’s efforts to learn more about what the Internet-based clinical trials search landscape offers a consumer who is looking to find clinical trials online. To these ends, we used a systematic analysis of the information and functions provided by websites that allow consumers to search for cancer clinical trials.

Methodology

Site Selection

A two-step process was used to identify and select for review the websites that offer a clinical trials search tool. First, the team that manages the National Cancer Institute’s (NCI) cancer.gov website recommended eight sites, including their Institute’s own, cancer.gov/clinicaltrials/search. The second step was to expand the site list by conducting an online search. Since most Internet users begin their searches with a search engine [20] and Google is the most widely used search engine [21], we expanded this list with Google searches using the search terms “cancer clinical trials” and “clinical trials.” Only site links on the first two pages of the Google findings for each search were considered because users rarely go beyond the first page of returned results [22]. Informational sites included within the Google search results were reviewed to see if they yielded additional clinical trials search tools.

Sites were included if they contained an English-language tool that allowed consumers to search for cancer clinical trials. Sites were excluded if the tool searched only studies available at one clinical center, if the links to the search tool were nonfunctional, if the search tool was not being currently maintained, or if the exact search tool was available on another site. For example, the American Cancer Society offers a search tool that is powered by EmergingMed, so only the main EmergingMed tool was included in this analysis.

Coding Schema

A preliminary review of the sites was conducted in April 2006 in order to create an inventory matrix of site features and functionality. Sites were then reviewed again, and the presence or absence of features and functionality was noted and entered into the inventory matrix. The site features examined in the analysis included basic search tools, advanced search tools, registration options, presentation of search results, and additional site content. See Table 1 for an overview of the coding schema.

Table 1.

Overview of the Coding Schema

Feature	Description
Basic search tool	Tools that allow the user to complete the entire search process from entering search parameters to receiving search results, including specific trial contact information, without requiring registration Coded as present or absent. If present, specific search parameters, such as cancer stage, were coded as present or absent Screen-based controls used to input search parameters, such as drop-down menus or scrolling lists, radio boxes, and/or text entry fields, were coded as present or absent Sites could have more than one basic search tool if these were discrete mechanisms for searching for trials that did not interact (e.g., a site might allow searching from both drop-down menus and open-ended text entry field).
Advanced search tool	Search tools that let users enter parameters in addition to those entered in the basic search function, without requiring registration. Coded as present or absent. If present, specific search parameters used and screen-based controls were coded as present or absent No site could be coded as having an advanced search without having a basic search. Advanced search tools might serve to refine the basic search.
Registration	Registration functions require the user to complete a registration process in order to use certain site functions. Coded as present or absent. If present, then specific registration functions coded as present or absent
Search results	Output of the clinical trials search. Coded as presenting results in tabular or narrative form. Specific content coded, such as trial phase or trial ID number To be coded as present, the specific content had to be clearly delineated in the results. For example, trial phase would be coded as present if there was a column entitled Trial Phase, but not if “Phase II” appeared in some trial titles. Option to refine results and/or to sort results coded as present or absent User’s next steps coded as present or absent Coded as present if they appeared on the same page as the trial results list or on the specific trial description. If a site recommended discussing potential trials with their physician on another general page but not on the trial results, the step of contacting a physician would be coded as absent.
Additional site content	Presence of general information about clinical trials coded as present or absent Presence of a medical dictionary or glossary coded as present or absent

Interrater Reliability

Interrater reliability was established by having a random selection of three of the selected sites coded by two coders. The three sites represented 20% of the sample, and included a total of 270 cells. This approach was chosen rather than coding a random sample of cells across the entire matrix because it allowed the coders to better mirror the in-depth review of each site that was necessary in coding the sites. A similar approach to establishing interrater reliability is reflected in the literature [23, 24]. The Kappa coefficient for interrater reliability was found to be .88, with reliability greater than .75 considered to show excellent agreement beyond chance [25]. Differences in coding were discussed and agreement was reached. The coding schema was modified to reflect the consensus and all sites were recoded according to the final coding schema.

RESULTS

Website Characteristics

All eight sites from the expert recommendation list were reviewed. The Google search returned six of these eight expert-recommended sites and yielded five additional sites. An additional site was identified during a review of informational sites within the search results (See Table 2). The resulting 14 sites included government, non-profit, and commercial sites. Half of the sites offered tools that searched only for cancer trials, while the others searched trials for a variety of health conditions, including cancer.

Table 2.

Overview of sites included in the website audit

Site Name and URL	Search Options	Search Fields	Results
Disease Focus: Cancer Only
Breastcancertrials.org 1 www.breastcancertrials.org	Registration	Contact information Age, Gender, Ethnicity, Education Trial location or zip code Type of cancer Daily activity level Current health status Diagnosis-specific questions Treatment undergone Other medical conditions Trial types user is interested in	ID number
Cancer Help UK 2 www.cancerhelp.org.uk	Basic Search	Type of cancer	Title Trial description
	Basic Search	Keyword search
	Advanced Search	Type of cancer Type of trial Trial location Trial phase
Cancer411 2 www.cancer411.org	Basic Search	Type of cancer Trial phase Trial location Age Drug Trial protocol ID number	Title
EmergingMed.com3 www.emergingmed.com	Registration	Contact information Age, Gender Trial location Type of cancer, Daily activity level, Diagnosis-specific questions Treatment undergone Other medical conditions Trial type preferences	Title Phase, Treatment modality Trial sponsor
National Cancer Institute 3 www.cancer.gov	Basic Search	Type of cancer Stage/subtype Type of trial Trial location	Title Phase ID number
	Advanced Search	Type of cancer Stage/ subtype Type of trial Status of trial, Trial ID number Location of trial Hospital/Institution, New trials, Type of treatment/intervention, Drug, Trial phase, Trial sponsor, Trial investigators, Lead organization
OntarioCancerTrials.ca 1 www.ontariocancertrials.ca	Basic Search	Keyword search	Title Trial ID number
	Basic Search	Type of cancer, Trial location or zip code, Type of trials, Drug
	Registration	Contact information Type of cancer
Trialcheck 3 www.cancertrialshelp.org	Basic Search	Type of cancer Stage/subtype Location of trial Age, Gender, Ethnicity Diagnosis-specific questions Current health status	Title, Trial ID number Trial location or distance Trial type
	Registration	Contact information
Disease Focus: Multiple Diseases
ClinicalTrials.com 2 www.clinicaltrials.com	Basic Search	Disease Type of cancer Trial location	Treatment modality Trial location Trial description
	Registration	Contact information Age, Gender Disease Type of cancer Previous trial participation
Current Controlled Trials: ISRCTN search and MetaRegister search 2 www.controlledtrials.com	Basic Search	Keyword search Keyword search	Title, ID number Trial sponsor
	Registration	Contact information
Medical Research Council Clinical Trials Unit 2 www.ctu.mrc.ac.uk	Basic Search	Disease; keyword search Disease; Type of cancer; Trial phase Type of trial	Title Trial ID number, Recruitment status Trial purpose
National Institutes of Health 3 www.clinicaltrials.gov	Basic Search	Keyword search	Title Condition Recruitment status
	Advanced Search	Disease Trial ID number Location of trial Hospital or Institution Type of treatment or intervention, Trial phase Trial sponsor Age Additional search terms
Thomson Centerwatch Clinical Trials Listing Service 3 www.centerwatch.com	Basic Search	Disease Type of cancer Trial location or zip code	Title Trial ID number Trial location or distance
	Advanced Search	Disease Type of cancer Trial location or zip code Keyword search
	Registration	Contact information Disease Type of cancer Trial location or zip code
TrialsCentral 2 www.trialscentral.org	Basic Search	Disease Type of cancer Trial location or zip code	Trial sponsor Trial location or distance
Veritas Medicine 3 www.veritasmedicine.com	Registration	Contact information Age, Gender Trial location or zip code Disease Type of cancer Other diagnosis-specific questions Treatment undergone Previous trial participation Other medical conditions	Title Trial location or distance

¹Identified by expert;

²Identified with Google search;

³Identified by both expert and Google search

Eleven sites used the basic search to allow users to enter search parameters without registering. Three of these sites with basic search tools offered two different search tools for the site’s database. Another site offered two basic search tools that each searched a different database. Four sites offered advanced search tools that allowed the user to enter additional search parameters to the basic search. Three of the eight sites with the registration feature required the user to register in order to receive any trial contact information.

Locating Clinical Trials Search Tools

Twelve of the 14 sites appeared within the first two results pages of a Google search using the terms “cancer clinical trials” and “clinical trials.” Four of the 12 sites listed on these Google results pages linked users directly to a page bearing the clinical trials search tool.

On the remaining eight sites, users had to follow a link from the site’s homepage to get to the clinical trials search tool. The terminology used on homepages to identify the search tool varied from site to site. Links to the search tools were identified in the following ways: “Clinical Trials” (2 sites), “Clinical Trials” (along with a description of the search function), “Trial Listings,” “Trial Search,” “Search for Trials,” “Search Studies” and “Browse Studies,” and “Start your search for a trial now.”

Entering Clinical Trials Search Parameters

The types of information that users needed to enter into the searches included a variety of screen-based controls: drop-down or scrolling list boxes, check boxes, radio boxes, and/or text entry fields. The search tools on most sites (11 of 14) required the use of at least two different types of controls. Only four sites limited the search tool to one type of control. Among these, the basic search tool on Clinicaltrials.com used only drop-down list boxes, while the basic search tool on NIH’s clinicaltrials.gov used an open-ended keyword search. CancerHelpUK and OntarioCancerTrials each offered two basic search tools: one search tool with drop-down list boxes, and the other with open-ended keyword searches.

The basic search on five sites requested that the user input only one piece of information, either the selection of a disease or type of cancer (for cancer-only search tools, such as at CancerHelpUK) or entering a keyword in an open-ended way (NIH’s clinicaltrials.gov). Eight sites, including NCI’s site, required the user to input between two and four search parameters, which typically included disease, cancer type, and geographic preference for trial locations. Two sites, Cancer411 and TrialCheck, asked the user to input six and eight search parameters with more specific information about stage of cancer, current treatments and medications, medical history, demographics, and/or trial preferences.

Four sites (NCI, clinicaltrials.gov, Thomson Centerwatch Clinical Trials Listing Service, and CancerHelpUK) offered an advanced search function. The advanced searches typically allowed users to input more specific information about the trial, such as trial identification number, trial phase, or sponsor.

Eight of the 14 sites reviewed had a registration feature (see Table 3). One, BreastCancerTrials.org, required users to register before allowing access to the search tool at all. Both Veritas Medicine and Emerging Med required a user to register in order to ultimately obtain trial contact information. Registration was optional on 5 other sites. In these cases, registration provided additional services to users. [Insert Table 3]

Table 3.

Registration Functions

Site Name	Functions
	Access to search tool	Access to trial contact information	Receive notification of new trials	Receive newsletters	Allow saved profile or searches
Breastcancertrials.org	X	X	X		X
ClinicalTrials.com			X
Current Controlled Trials: ISRCTN search and MetaRegister search				X
Trialcheck			X	X	X
EmergingMed.com		X	X		X
OntarioCancerTrials.ca			X		X
Thomson Centerwatch Clinical Trials Listing Service			X	X
Veritas Medicine		X	X	X	X

Using Clinical Trials Search Results

Regardless of search mechanism, results were generally returned in a tabular format with a variety of content across sites. Only CancerHelpUK delivered results in a narrative format. The results tables were organized in a variety of ways: by percentage match to search criteria, trial phase, trial title, location, trial sponsor, or trial purpose (e.g., treatment or prevention). However, the organizational strategy was not readily apparent on most sites and could not be ascertained on five sites. Further, sites often did not make clear what database of trials they were searching, so users would not know if the search on one site possibly duplicated the search on another site or how wide the potential universe of trials could be.

Six sites offered a mechanism to narrow or refine the search. In these cases, the user might: 1) arrive at the site’s advanced search tool (NCI, Thomson Centerwatch, CancerHelpUK), 2) be given the option to add a keyword search (clinicaltrials.gov), 3) return to the original search form (Trialcheck), or 4) be offered help with matching the user to trials through either telephone contact or by having the user enter more information into an online profile (Emerging Med).

Just three sites (4 search tools) allowed the user to sort the search results, with only TrialsCentral.org providing sorting instructions directly on the results page. On NCI’s site, the user needed to follow a Help with Results link to find instructions on how to sort. Results from Current Controlled Trials could be sorted by selecting a criterion (e.g. trial title, relevance, study ID number) from a pull-down menu entitled Results Order on the top of the table.

Specific trial descriptions were accessed by following links from the results tables. Generally, descriptive information included the trial title, trial description or purpose, and eligibility criteria. All but two sites provided trial contact information. Both the trial results and the trial descriptions tended to be dense with complex terminology. Two exceptions were seen in the CancerHelpUK site, where an extensive effort was made to convey trial information in plain English, and the NCI site, which included both a patient and a professional description of the specific trials. The depth of the descriptions varied across sites and even between trials found on the same site. Sometimes following the link for a trial description moved a user from the trial search website to the website of origin for that specific trial.

The sites also differed in how clearly the trial descriptions conveyed what a user’s next steps would be. Twelve sites provided trial contact information in the description, including the name, address, phone, and email of the primary investigator, many with the implicit assumption that the user would then contact the trial. Three of these sites (NCI, Ontario Cancer Trials, & Clinical Trials Unit of the Medical Research Council) also advised the user to discuss search results with their physician to help determine whether a trial was appropriate. Veritas Medicine, BreastCancerTrials.org, and Emerging Med offered to initiate contact with the investigators for the user.

Additional Site Content

All but two sites (Cancer411 and Current Controlled Trials) provided users with general information about clinical trials such as what trials were and the benefits and risks of participation. Ten sites provided a dictionary or glossary, but these were not easily found on all sites, and the quality was variable.

Summary of Search Process

Locating search tools was relatively simple using a popular search engine, although an individual’s success may be dependent on the actual search terms entered. In some cases, locating the tool may be challenging if the user must then find and follow a link from the homepage. Additionally, the links to search tools were not labeled consistently on different sites, further complicating the search.

The search tools themselves showed considerable variability from site to site. Some allowed the user to type in their own search query, while most offered a parametric search tool that conducted the search based on multiple criteria, such as type of cancer or geographic location. The user needed to be prepared to enter their general diagnosis into the simplest parametric search tools and very specific medical information and/or trial information into the more complex tools. Some sites required users to register before they could view trials. Mastering the features and functionality of one site did not necessarily mean that a user would be successful on other sites because of this lack of consistency between search tools. Users would need to learn how to use the unique search process for each site they might visit.

Both the trial results and the trial descriptions tended to be dense with complex terminology. Users might need to sift through trial results that could number in the hundreds. These results can contain diagnostic terminology like the stage and type of the cancer, treatment terminology such as the names of chemotherapy drugs, and research terminology such as “trial phase” and “random assignment.” Determining how to sort or refine search results was not simple, and this function showed considerable variation between the sites. A steep learning curve may be associated with understanding all that a specific search tool can do. Finally, if trials of interest are found, users in most cases must take the initiative to bring promising trials to the attention of their physician or to contact the trial directly.

DISCUSSION

This study sought to determine the characteristics of available search tools in order to understand what consumers will encounter when they search online for clinical trials. Findings from the audit showed that it was relatively easy to locate a search tool, but actually using the search tools was a more complex endeavor. The search interfaces on different sites showed significant variation and, consistent with previous studies [15,16], many sites were dense with medical and research terminology. Therefore, clinical trials searchers will need to be skilled users of the Internet, knowledgeable about the diagnosis and treatment history, not intimidated by medical and research jargon, and have the perseverance to sift through search results that could potentially number in the thousands.

Current usability guidelines provide an additional mechanism for examining clinical trials search tools. These guidelines offer recommendations for both the search interface and the search results. However, readily available guidelines generally pertain to helping a user locate information on a website, rather than to these more complex search tools designed to match specific user input to specific clinical trials within a database of clinical trials.

The Research-Based Web Design & Usability Guidelines [26], developed by the U.S. Department of Health and Human Services, recommends that users be provided with simple search functions (Guideline 17.6), search templates (Guideline 17.9), and hints to improve search performance (Guideline 17.8). A simple type-in text field is the recommended interface for simple search functions [27], despite studies that show that formulating a search query can be difficult for users [28, 29]. Nielsen [30] reported that since searching has become such a prominent part of users’ experience with the web, they have developed strong mental models for how searches should function. Users expect searches to have a box in which they can type words, a button labeled “search” that they click to run the search, and a list of results that is linear, prioritized, and appears on a new page [30].

Given the complexity of terminology associated with a cancer diagnosis and potential treatments and trials, a simple text search may not be enough for many users. Clinicaltrials.gov was the only American site that offered an open text search box as the initial search strategy. The other sites offered parametric searches, which contain an array of fixed choices from which a user can select search parameters. Another usability resource, Usability First, cautions that users may constrain their search too much when using parametric search tools and get no results [31]. It is not clear whether open-ended simple searches or parametric search options will be more successful in helping users formulate a search query when searching for clinical trials.

Offering registration on the search tool provided users with additional features that could aid their search. However, other research suggests that requiring registration may add too much of a burden on users, especially those coping with a serious illness like cancer [32]. Although the Research-Based Web Design & Usability Guidelines [26] do not specifically address website registration issues, they do recommend several strategies to optimize the user experience including reducing the user’s workload (Guideline 2.4).

With respect to search output, Usability.gov [26] recommends ensuring usable results (Guideline 17.1) and displaying information in a directly usable format (Guideline 2.8). Similar to Nielsen’s report [30], Quesenbery [33] found that users expected results to be organized with the best matches first. The search results on the sites examined in this audit were organized in many different ways, and, in some cases, the organizational strategy could not be determined. Organizational strategies that do not match user expectations may pose another problem for trial searchers, who might wrongly assume that the trials presented first are the best match for them.

Several limitations to this study should be noted. The website audit was limited by the use of researcher-generated search terms to locate clinical trials websites. The general public may use different search terminology and their results may then differ. Further, the Internet is capable of rapid change, and some of the reviewed websites may have been modified since this review.

Strengths of this research include the examination of the current clinical trials landscape and what potential Internet users will likely encounter. The audit showed that many sophisticated steps are required for a successful Internet search.

This study is important because these findings can be used to systematically examine online clinical trials search tools to determine whether and which clinical trials search tools actually meet the needs of those who are looking for clinical trials. Specific questions include what search tool interfaces users prefer, whether users are more successful with an open-ended keyword search or the more restrictive parametric search tools, how users perceive registration requirements, whether they are able to locate trials of interest from within search results, and if so, whether they can ascertain what their next steps might be.

This information will also enable the identification of quality tools that clinicians can recommend to their patients and even use themselves to review clinical trial research. Patients new to clinical trials may benefit from using basic search mechanisms and background materials to familiarize themselves with the types of trials that are available and the clinical trial process. However, advanced search functions and registration-based search tools will likely be more useful for identifying trials that match patient needs. Given the complexity of the search process and the need for timely identification of trials, clinicians could help their patients by becoming familiar with these tools and facilitating their use. They might also designate a staff member to serve as a resource person to step the patient through the search process.

Developers of these tools, including NCI and other health organizations, can use this and subsequent research to improve the search process and clinical trial recruitment. Besides getting input from cancer patients to simplify and enhance their search options, developers could share the tools with clinicians and their staff members to determine the best ways to integrate clinical trial search procedures into the health care encounter. Ultimately, by ensuring the usability of online search tools for current and prospective users, we can expand the options for those facing cancer.