Pilot Study of a Point-of-use Decision Support Tool for Cancer Clinical Trials Eligibility

Abstract

Many adults with cancer are not enrolled in clinical trials because caregivers do not have the time to match the patient's clinical findings with varying eligibility criteria associated with multiple trials for which the patient might be eligible. The authors developed a point-of-use portable decision support tool (DS-TRIEL) to automate this matching process. The support tool consists of a hand-held computer with a programmable relational database. A two-level hierarchic decision framework was used for the identification of eligible subjects for two open breast cancer clinical trials. The hand-held computer also provides protocol consent forms and schemas to further help the busy oncologist. This decision support tool and the decision framework on which it is based could be used for multiple trials and different cancer sites.

Despite national attention to breast cancer, less than 10 percent of the approximately 180,000 women diagnosed each year with breast cancer are enrolled in therapeutic clinical trials.^1,2 If more women participated in trials, they would benefit by receiving state-of-the-art therapy, and improvements in the prognosis and quality of life for all women affected by this disease could be accelerated. In addition, greater participation of women would give clinical trials greater external validity.

Patient³ and physician⁴ factors can be barriers to the enrollment of women in breast cancer therapeutic trials. Patient factors include lack of access to a health center offering clinical trials, economic and social barriers, and attitudes and beliefs.⁵ Physicians may fail to offer clinical trial participation to patients for a variety of reasons, including lack of time.^6,7 Physicians participating in a busy oncology practice may feel that they do not have the time to efficiently identify eligible study subjects, or they may simply forget to offer and enroll patients in open trials.⁸ Completing these tasks requires a level of access to clinical trial information that does not exist in most practice settings. Typically, clinical trial documents are placed in binders in a central location in the practice (▶) or they are simply not available on site. To determine whether new patients may be eligible for an open trial, physicians may need to open the binder and read through the eligibility sections of one or more protocol documents. This activity interrupts the flow of patients for clinical practitioners who are increasingly pressed for time, and constitutes a substantial barrier to trial enrollment.

Figure 1.

Protocol documents. These are placed in large binders on a shelf in a conference room.

We hypothesized that a portable computer-based decision support tool designed to rapidly screen patients for eligibility for breast cancer clinical trials would overcome this barrier. We envisioned a system that would rapidly point the clinician to a specific trial or inform the clinician that the patient was ineligible for all open trials. This would be accomplished by matching a small set of clinical data to eligibility criteria of open trials.

We report on the design of a hand-held decision support tool as our first step in testing these hypotheses. We refer to this tool as DS-TRIEL (Decision Support for TRIal ELgibility). We highlight lessons learned and issues that future systems should address.

Background

Some attention has been paid to creating solutions to the problem of identifying subjects who may be eligible for open cancer clinical trials. For example, physicians may use the National Cancer Institute's PDQ (Physician Data Query) clinical trial search form.^9,10 This useful database tool retrieves protocols by diagnosis, phase, modality, sponsorship, drug, and geographic location. However, it does not point the user in the direction of a specific protocol. To accomplish this, the physician must match the patient's clinical data to the specific eligibility criteria listed in various protocol documents.

A few reports describe decision support systems designed to help physicians match patients with human immunodeficiency virus (HIV) to therapy-related research projects. Tu et al.¹¹ used a qualitative heuristic approach, while Ohno-Machado et al.¹² used Bayesian belief networks to manage uncertainty associated with clinical data missing at the time of eligibility determination. In essence, given limited clinical data, these systems provided the physician with a probability that a specific patient would be eligible for a clinical trial. A similar, recently reported Bayesian network approach¹³ helps those assigning patients to breast cancer clinical trials. Substantial investment is required to construct, train, and maintain these systems. A software module approach that allows for acquisition and maintenance of protocol knowledge has been described.¹⁴ This sophisticated and highly developed system directly addresses the issue of maintenance when new protocols need to be added to a system. It has the added advantage of being a general-purpose system, as it has been used in several clinical domains.

We took a slightly different approach, for several reasons. First, the probabilistic systems mentioned here help the clinician deal with missing clinical information by providing estimates of the probability that a patient will be eligible for a given trial once all the appropriate clinical information is known. In oncology practice, missing clinical information is not a substantial barrier to clinical trial enrollment. Data such as histologic findings, stage, tumor size, and lymph node involvement are almost always available at the first medical oncology visit. Instead, the clinician needs to be quickly pointed to a limited number of protocols whose major eligibility requirements exactly match the disease characteristics of a particular patient. For this reason, we developed a decision support tool based on a deterministic model with a simple two-tier decision framework. Second, the clinicians engaged in this effort wanted a testable pilot system as soon as possible. This dictated the development of a simple data model. Finally, we focused on developing a portable pilot system, which could be carried by medical oncologists as they provide care.

Clinical Setting

The Indiana University Comprehensive Cancer Center's (IUCC) Breast Cancer Program wanted to increase enrollment in breast cancer trials. To address this aim, we collaborated to develop a portable electronic tool to help identify potentially eligible subjects. Patients in the IUCC program are evaluated in one of two settings. One involves a multidisciplinary ambulatory program in the Cancer Pavilion of the IUCC. Each new patient is evaluated by a team consisting of a fellow in medical oncology, a faculty medical oncologist with specialized interest in breast cancer, a breast cancer surgeon, and a nurse coordinator. The other is a community-based satellite practice of the breast cancer program, where patients are evaluated by a medical oncologist with specialized interest in breast cancer along with a nurse coordinator.

Patients are referred to the IUCC Breast Cancer Program from the university-affiliated hospitals and other health care providers. Patients usually come for a first visit with significant clinical material to confirm histologic diagnosis, stage, and disease status. Using this clinical information, the clinic nurse coordinator performs an initial screening of patients for clinical trial eligibility. Before the development of DS-TRIEL, this screening consisted of documenting variables such as age, histologic diagnosis, stage, and major events in the medical history and entering them on a medical summary form found at the front of the clinic record. The nurse used his or her knowledge of the open trials to indicate to the medical oncologist whether the patient might be eligible for any open trial. The medical oncologist further evaluated the patient for trial eligibility by filling in gaps in the clinical information. The efficiency and reliability of these screening steps generally depended on the familiarity of the nurse and physician with the eligibility requirements of the specific trials.

Routinely, only a subset of the clinical information required to determine final eligibility for a given trial is available at this initial visit. The usual approach to this issue is to request additional laboratory testing to determine final eligibility, since most trials include restrictions of hematopoietic, renal, and hepatic functions. The results of this blood work are compared with the study requirements by referencing the protocol document. If the patient remains potentially eligible for inclusion in the study, any remaining clinical and historical data that are needed to determine eligibility and were previously overlooked are obtained by reviewing the eligibility checklist criteria with the patient. Assuming the patient remains eligible after completion of the eligibility checklist, the trial and its objectives are explained and informed consent is obtained. This step often requires a review of the protocol document for one or more open trials. Our information system was designed to make this screening and enrollment process more efficient.

During the development of the system we describe, there were two open clinical trials for breast cancer. The “double docs” study was enrolling certain women in whom breast cancer had recently been diagnosed and was designed to assess the efficacy of a new drug combination, docetaxel and doxorubicin. The other study was enrolling women who had been previously treated for early-stage breast cancer. It was designed to assess the efficacy of Marimastat, a new metalloproteinase inhibitor, to be given following primary adjuvant chemotherapy. The specific eligibility requirements for these two clinical trials are summarized in ▶. There is no eligibility overlap with these two trials.

Table 1.

Eligibility Requirements for Breast Cancer Studies

Eligibility Criteria	“Double Docs” Study	Marimistat Study
Major	Stage II or III disease	Lymph node negative
		Tumor size greater than 1 cm
		Estrogen receptor negative or Lymph node negative
		Tumor size greater than 2 cm
		Estrogen receptor negative or Lymph node positive
		Stage II or IIIA disease
	Measurable disease
	No prior chemotherapy	Prior adjuvant cyclophosphamide and doxorubicin treatment or current adjuvant tamoxifen treatment
Minor	Normal hematopoietic function*	Normal hematopoietic function*
	Normal liver function*	Normal liver function*
	Normal renal function*	Normal renal function*
Note: The “double docs” study was designed to assess the efficacy of a new drug combination, doxetaxel and doxorubicin, and the Marimistat study to assess a new metalloproteinase inhibitor.

^*Normal function was defined a priori for each trial by specific laboratory criteria.

Design Objectives

The primary design objective (▶) was the efficient and reliable identification of patients with breast cancer who are eligible for clinical trials. This was to be done without impeding workflow. In fact, it was expected that such a system could improve workflow by eliminating the need to repeatedly reference protocol documents stored away from the point of patient contact (usually in a staff conference room). A second objective was to embed additional functions related to trial enrollment into the system to make it more attractive to users. These functions made available electronic versions of the informed consent statement, the protocol schema, and a “to do” list. Finally, we wanted development time for this pilot project to be short. Lessons learned from this pilot could then be incorporated into an improved decision support system and formally assessed.

Table 2.

Objectives of System Design

Make trial subject identification efficient and reliable

Improve workflow Be Portable

Provide additional functions

Informed consent statement Protocol schema “To do” list

Provide simple decision framework

Short development time Easy maintenance

System Description

Hardware

Before developing this system, we performed time-motion studies at the IUCC to analyze work processes. We found that 25 percent of physician and nurse time was spent walking from one place to another (P. P. Breitfeld, unpublished data). On the basis of these data, we felt that a clinician would find a decision support tool used at the site of patient encounter advantageous. In our setting, however, there are no computer terminals in patient examination or interview rooms, and networked stations are found only in central conference rooms, which are remote from patients. Thus, if the decision support software were to reside on a server and be available through our networked computers, physicians and nurses would still be unable to access clinical trials information at the point of patient contact. Such a system would be unlikely to improve workflow. Thus, a portable system would have the potential to make the decision support tool available at its most efficient point of use.

The hardware options for point-of-use clinical information systems have steadily grown in number. When we planned this program in early 1997, available options included tablet-style computers that featured a pen-based interface and varied from a notebook size to a breast-pocket size. Examples included the Palm Pilot (3Com, Santa Clara, California), the Fujitsu Stylistic (Fujitsu Personal Systems, Tokyo, Japan), and the Apple Newton Message Pad. Another option included small hand-held computers, with or without a pen interface, that have a smaller-than-standard keyboard. These devices have various operating systems, including the Apple Newton system, Microsoft Windows CE, and even Microsoft Windows 95.

We wanted a device, small enough to fit in a pocket, that could give rapid feedback about trial eligibility for patients once information was input. It had to have a readable screen and be able to print selected output on network printers. We reasoned that the need for speed would initially dictate a stand-alone system able to run a programmable relational database. Our goal, however, was to design a system that could be hosted on any hand-held device capable of running Microsoft Access or a similar relational database software.

Given these user requirements, we choose the Libretto (Toshiba, Tokyo, Japan). It weighs less than 2 lb, can fit in the side pocket of a clinician's white coat (▶, top), and has the functions of a desktop computer, since it can run Windows 95. Although it is small, its keyboard is usable for short stretches of work and its screen is clear (▶, bottom). Its battery's life span allows use for this application for a full clinic day with recharging overnight. Finally it satisfied the requirement that it support a programmable relational database such as MS-Access.

Figure 2.

The Toshiba Libretto device easily fits in a whitecoat pocket (top), and although it is small compared with a standard desktop computer, its screen is clear (bottom).

Software

The overall design of DS-TRIEL was based on representing the relationships between breast cancer clinical trials and their eligibility criteria. The objectives of the model were to minimize the amount of clinical data entry necessary to determine patient eligibility and to minimize the programming effort required to accommodate new clinical trials. The model is illustrated in ▶. In essence, the model defines categories of data required to determine clinical trial eligibility (categories I and II, see below), categories of clinical trials, and specific relationships between the two.

Figure 3.

Clinical trials model. Data required for eligibility determination (I and II, far left) and clinical trials are categorized A through F (middle left). There are defined relationships between instances of category I data and trial subgroups A through F. Some of the relationships between instances of category I data and neo-adjuvant trials are indicated (middle right and far right).

Data required for determination of clinical trial eligibility are organized in a two-level hierarchy (▶). The first level (category I) was defined to minimize the number of data elements needed to maximize matching to a particular trial category. The second level (category II) concerns all other data used to determine eligibility and serves to determine final eligibility. We defined category I data by at least four characteristics. In addition to maximizing the probability of matching patients to a single trial category or to none, category I data are known at the first oncology visit and are constant over the short term. Importantly, instances of category I data segregate to subgroups of trials. This aspect of the model greatly reduces programming effort when new clinical trials in a given subgroup become available for enrollment of subjects.

Table 3.

Hierarchy of Eligibility Data

	Category I Data	Category II Data
Characteristics	Maximizes matching	All other data, not in category I
	Are common across trials
	Are known at first visit
	Are constant over short term
Objective	Defined to minimize initial number of data elements	Defined to allow definitive determination of trial eligibility
Examples	Prior chemotherapy or radiation treatment	Serum creatinine level
		Platelet count
	Prior surgery
	Histologic findings
	Tumor size
	Nodal status
	Presence of metastasis
	Estrogen receptor status

The set of all breast cancer clinical trials can be subgrouped (▶, far left and middle left). Each subgroup of trials is designed to address important issues for a defined group of patients. For example, patients with measurable disease are eligible for neoadjuvant clinical trials that measure the response rate of breast cancer to new combinations of chemotherapy agents. Well-defined and unique combinations of category I data define the patients of interest for a particular subgroup of trials. ▶, middle right and far right, illustrates the relationship between instances of category I data and neo-adjuvant chemotherapy trials. Therefore, instantiation of category I data places a given patient into a single trial subgroup. In our setting, the “double docs” trial was an example of a neo-adjuvant trial and the “Marimistat” trial was an example of an adjuvant post-primary therapy trial. The categorization of clinical trials makes updating DS-TRIEL easier when trials close and new ones open.

A subgroup may include more than one trial. When it does, each trial is assigned a priority. Assignment to a single trial is accomplished by a weighted randomized scheme that reflects an established relative priority of the trials or, if all subgroup trials are of equal priority, by a completely randomized assignment. In this way, the model allows for an individual patient to be potentially eligible for more than one open trial but focuses the oncologist on a single trial. Instantiation of category II data determines final eligibility for the trial selected by category I data. If the subject proves to be ineligible and if the subgroup includes more than one trial, the subject can be assigned to the next trial on the priority list. Again, instantiation of category II data determines final eligibility. Instantiation of category II data does not move a patient out of a subgroup but may move a patient to the next lower priority trial in the subgroup.

The two-node decision tree that evolved from this model is illustrated in ▶. The first node requires instantiation of category I data, and the second requires instantiation of category II data. Although conceptually simple, five iterations and 30 hours of effort among the expert clinician, medical informatics specialist, and programmer were required to fully develop this model.

Figure 4.

Decision framework. Determination of trial eligibility is accomplished in two steps. The first decision utilizes category I data (represented by the circular chance node “I”) and leads to potential eligibility for a single trial category or ineligibility for all. The second decision utilizes category II data (indicated by chance nodes “II_A through II_F”) and leads to a final determination of trial eligibility. If a given category includes more than one trial, a weighted randomization scheme determines the most appropriate trial assignment. Category II data are then applied to eligibility criteria for a single trial.

DS-TRIEL was developed with input from clinicians throughout the process. The user interface is divided into three sections (▶). The first section involves registering the patient in the database by inputting the patient's name, date of birth, medical record number, and site of clinical encounter (▶). Such data elements also may be downloaded to the Microsoft Access database from a central database containing a list of scheduled patients. The second section involves a series of screens for entering structured clinical data that are critical for assessing breast cancer clinical trial eligibility (see Figures ▶ and ▶ as examples). This small set of data, referred to as category I, includes evidence of tumor resection or of adjuvant chemotherapy or radiation therapy, tumor histology and size, nodal or distant metastasis, and estrogen receptor status (▶). These data can be entered either by pointing and clicking or by striking the keyboard. Once completed, this section ends with a message screen that informs the user about all potential clinical trials for which the patient may be eligible (▶).

Table 4.

Features of DS-TRIEL System

Patient registration

Structured clinical data entry

Output:

Eligibility checklist Study consent Study schema “To do” list

Figure 5.

Patient registration screen. This is the first screen the user encounters. The patient information is entered if it is new or if the patient has been selected from a menu of previously registered patients.

Figure 6.

Structured clinical data entry screen. This is the first of a series of four pages that ask the user to input specific clinical data.

Figure 7.

Structured clinical data entry screen. This is the second of a series of four pages that ask the user to input specific clinical data.

Figure 8.

Structured clinical data summary screen. This screen can be viewed after all data have been entered. It can be printed for placement in a paper record.

Figure 9.

Eligibility message. This message appears once the clinical data have been entered and confirmed.

Once a patient has been declared potentially eligible, the user has a choice of several different types of output (▶). The study eligibility checklist can be viewed (▶) and, once completed, can be printed. If an answer to a question on the checklist disqualifies a patient from the study, a message that the patient is ineligible is given immediately and the program reverts to the registration screen. If, on the other hand, the patient remains eligible, a copy of the study consent can be printed for distribution to the patient. The user can view and print the study schema so that the clinician can see the treatment plan for the study and give a copy to the patient (▶). The user can view and have printed a list of tests that need to be completed and queries that need to be answered to complete the study eligibility checklist and begin protocol therapy (▶). We refer to this as a “to do” list.

Figure 10.

Output. This screen offers the user several output options, including the consent form, and a study “to do” list (study tasks to be completed), and a study schema.

Figure 11.

Eligibility checklist. This allows the user to view and complete the eligibility checklist online.

Figure 12.

Study schema. This allows the user to view and print the study schema as it appears in the protocol document.

Figure 13.

Study tasks. This screen outlines tests not yet done and queries not yet answered that are needed to satisfy all eligibility requirements (left side) and all tests needed before the patient begins protocol therapy (right side).

Connectivity

We designed the hand-held computer to be the means of data entry and to house the logic of the eligibility decision tree. In our system, the hand-held computer thus acts as both “client” and “server,” and connection to a central server is not needed. In our prototype, connection only to office printers is needed. This is accomplished easily with the Toshiba device, since the “dock” is permanently connected to the printer and the computer can be easily “docked” whenever printing is needed. We anticipate future development of wireless and Web-based versions of this system.

Preliminary Evaluation: Lessons Learned

We pilot-tested DS-TRIEL in the practice of an academic medical oncologist with a specialized interest in breast cancer. The medical oncologist and four nurses performed the testing. The system was easily usable by them. It was so well accepted that when the pilot testing finished, they requested that DS-TRIEL remain on site and that newly opened trials be coded into the system. Most clinical data entry, usually the most time-consuming part of such systems, can be quickly performed by selecting menu items. We found that it took less than one minute for the nurse or physician to enter category I data and receive the initial message regarding potential trial eligibility. We also entered category I data from ten candidate patients. In all ten cases, DS-TRIEL provided eligibility assignments consistent with determinations made independently. We anticipate a formal analysis of the impact of this system by performing time-motion studies of users in clinical settings with and without DS-TRIEL.

The development and pilot testing of this system has taught us several lessons. First, we learned that a two-level hierarchic decision framework can be used to design a useful breast cancer clinical trial decision support tool. However, decisions about placing data into category I or II “bins” and classification of trial subgroups require input from an expert clinical investigator experienced in clinical trial eligibility criteria. Second, the system was more valuable to clinicians when it performed functions in addition to determining clinical trial eligibility. For example, clinicians suggested that the system provide a “to do” list that served to focus work that needed to be done before study therapy was begun. Third, in pilot-testing the system, we found that nurses used the system more often than physicians did, primarily because nurses have first contact with patients and patient data. Thus, the system provided nurses with unambiguous decisions on patient eligibility, without the need to interrupt the physician or reference remote protocol documents. Finally, since all patients can be registered in the system, a registry of all patients is created and further study of factors that predict which patients are not enrolled or even offered clinical trial participation is made more feasible.

We discovered certain limitations. For example, most users would prefer a computer that did not require time to “boot up.” For data entry, the pointing device of the Libretto was acceptable but, compared with a desktop mouse, took extra time to use. Consequently, we programmed the application so that single keyboard strokes would substitute for pointing and clicking. Our model and decision framework could hamper the ease of extending this system to other clinical domains.¹⁵ For example, had we employed a software module that generated problem-solving behaviors, extension of DS-TRIEL to other clinical domains would be simpler. For example, category I and II data and trial subgroups would need to be redefined if this model were applied to cancers other than breast cancer. Nevertheless, for any clinical domain, many clinical trials can be assessed and then placed into appropriate trial subgroups. Similarly, their eligibility data can be classified as category I or II data using our model. Finally, the stand-alone nature of the prototype requires that protocol updates be distributed to all portable computers. In our pilot testing this was a minor problem, since we tested the system at only two sites. Widespread use of DS-TRIEL would be more efficient if the database were maintained in a centrally assessable server.

Conclusions and Future Directions

The World Wide Web contains very useful, authoritative resources for cancer information.¹⁶ In addition, knowledge-base tools that focus on cancer include those that help investigators in the design and construction of protocol documents¹⁷ and those that help clinicians adhere to protocol study and therapy requirements.¹⁴ Nevertheless, DS-TRIEL makes a contribution to the field of cancer trial informatics as a decision support tool that helps identify subjects who are eligible for open clinical trials. Specifically, it differs from National Cancer Institute's PDQ in that it provides electronic matching of an individual patient's clinical data to study eligibility criteria. This eliminates the need for the research nurse or physician to remember eligibility requirements, look them up in clumsy binders, or view them one by one on a computer screen.

Our system also makes a contribution to the field by demonstrating that the process of determining clinical trial eligibility can be organized into a two-tier hierarchic model. Whether this framework will be useful for cancers other than breast cancer remains to be tested. Nevertheless, the practical application of this model and decision support tool has implications for the design of future therapeutic trials in cancer. For example, we found that the eligibility criteria for hematologic, renal, and liver function varied arbitrarily from study to study. Standards for these entry criteria, with allowance for modifications based on appropriate evidence, would simplify eligibility determination. More important, the use of a general framework such as ours could also influence investigators designing new trials by reducing arbitrary variation in major eligibility criteria for a given category of patients.

Further development and testing of this point-of-use system should include general oncology practitioners. These busy medical oncologists care for a wide variety of cancer patients, and many wish to enroll their patients in available open trials. However, it is unrealistic to expect them to be familiar with all open trials or to take the time to sort through multiple trial protocols to determine whether each patient might be potentially eligible for a study. Our system should help overcome these barriers and increase enrollment of private-practice patients in open trials. Future systems should be able to sort through protocols for other common cancers, such as those of the lung, colon, and prostate. Moreover, these systems should be able to effectively deal with instances where a patient is eligible for more than one open clinical trial. In these cases, simple logic would allow for prioritization of enrollment into certain protocols or assignment of patients to a trial using a weighted but random enrollment scheme.

Once such a system has been developed, patient registration and clinical data entry might be accomplished by capturing relevant data from a computerized medical record. We expect to explore this possibility with the Regenstrief medical record system,¹⁸ a computerized patient record developed at the Indiana University School of Medicine. This electronic data capture would further reduce the work and increase the accuracy and timeliness of data entry for a busy medical oncologist. The portable nature of our design could be preserved with wireless communication between a point-of-use portable client and a central server housing the electronic medical record.

Finally, a Web-based front-end for this application might be desirable. In this case, patient data would be entered without patient identifiers, and any medical oncologist with an Internet service provider and a Web browser could access the application, located on a central server, and be rapidly pointed toward a clinical trial for a given patient. Such a system has at least two advantages. First, the centralized database is easier to maintain, since protocols are amended and updated as needed.¹⁹ Second, it allows for the centralization of clinical trial enrollment, an important consideration for multicenter national trials. Today, this approach might require operation on a desktop or bulky laptop computer wired to the Internet. Recent advances in hand-held computers, such as Internet browser capability and wireless communication, will soon allow for portability in the clinical setting, an important feature of our prototype.