Abstract
Use of major eligibility criteria is a popular but unstudied folk practice for improving patient screening efficiency for clinical studies. This mixed-methods research study derived the desiderata for major eligibility criteria in breast cancer clinical trials. We randomly selected thirty interventional breast cancer clinical trials conducted at The New York-Presbyterian Hospital on the Columbia University Medical Center campus to create training (N=20) and testing (N=10) datasets. We utilized the Think-aloud protocol to gauge how clinical researchers identify and use major eligibility criteria to prescreen patients for clinical trials during an audio-recorded interview. A focus group session was held to understand the current prescreening process and investigate how it could be optimized to maximize recruitment rates. Using the grounded theory method, we annotated transcriptions to discover user rationale and desiderata behind major eligibility criteria in breast cancer clinical trials, which were later evaluated in a follow-up survey.
Introduction
Recruitment to clinical trials remains the biggest barrier to clinical and translational research. Most clinical trial studies have dozens of complex inclusion and exclusion eligibility criteria. Therefore, when using these eligibility criteria for subject screening, a common and cost-effective practice adopted by clinical researchers is to start with a small set of “major eligibility criteria”. If a patient does not satisfy any major criterion, there will be no need to waste the time to go over the complete eligibility criteria list. Moreover, many clinical research coordinators know that a lot of minor eligibility criteria are either irrelevant or rare among the target population so that the information gained from those minor criteria returns minimal value for screening eligible patients. Therefore, they focus on using major eligibility criteria to maximize information gain during prescreening processes.
Understanding the definition and characteristics of major eligibility criteria has three foreseeable benefits: (1) facilitating automated selection of major eligibility criteria for efficient electronic clinical trial prescreening using electronic patient data; (2) enabling cost-effective analysis of existing eligibility criteria; and (3) enabling understanding of the prioritization behaviors of clinical researchers to inform the design of advanced clinical trial screening methods. However, to the best of our knowledge, little is known about the selection of major eligibility criteria for clinical trial prescreening. With a focus on the important disease area of cancer, in this study our research questions are “is there a common definition for major cancer clinical trial eligibility criteria?” and “if yes, what are the characteristics of these major eligibility criteria?” In this study, we define a desiderata for major eligibility criteria of clinical trials as well as provide examples of standardized major eligibility criteria using a mixed-methods approach using qualitative and quantitative measures with clinical researchers at Columbia University Medical Center1. The study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects, and was approved by the Columbia University Medical Center Institutional Review Board.
Background
1. Barriers to recruitment and the need for prescreening
Slow clinical trial recruitment is a significant obstacle to medical discovery2. Recruitment obstacles can lead to trial completion delays from one to six months for most clinical trials. Automating a significant portion of the patient screening process could reduce recruitment delays by identifying potentially eligible patients. The prescreening of patients for clinical trials requires use of study specific predefined inclusion and exclusion eligibility criteria. The lack of standardization for eligibility criteria creates obstacles to comparing trials to one another3–4. As a result, patient screening is a complex multi-step and multi-user process that requires significant and expensive human time and effort5. Moreover, the complexity of eligibility criteria can cause potentially eligible patients to be overlooked6–7. Information systems have the potential to decrease delays in clinical trial recruitment and minimize human resources that manually review records for prescreening. Weng et al. showed that a real-time screening alert for a single clinical trial before a patient clinical encounter improved the rate of recruitment over a twelve-month period by identifying patients that are “potentially eligible” for full screening2. The challenge is to prescreen a patient for an entire portfolio of clinical trials.
2. Related work
Most prior efforts have focused on the standardization of eligibility criteria with expressive ontologies. For example, the Eligibility Rule Grammar Ontology (ERGO) provides a formal representation for eligibility criteria8. ERGO is not sensitive to the complexity of criteria and does not represent the stability of eligibility data elements or support fuzzy representations such as “potentially eligible”. A related effort called the eligibility criteria extraction and representation (EliXR) extended ERGO by providing richer semantic information and decomposing complex eligibility criteria into more meaningful semantic segments9. EliXR-Time further extended EliXR by supporting temporal representations10. For example, a temporal constraint might require a medication washout period of three weeks. Such efforts have focused on rich semantics and the full expressiveness of the criteria, though none of them has been widely adopted yet11 or have generated real-world impact on clinical trial recruitment processes. In this study, we took a novel approach to this problem by addressing a real user need for classifying major eligibility criteria, which has been observed in folk practice of clinical trial prescreening and has remained largely unstudied.
3. Challenges of cancer clinical trial recruitment
Unique challenges exist for cancer clinical trials. The knowledge of cancer is increasing at an unprecedented rate along with its understood complexity12. Patients that can be recruited for cancer clinical trials vary widely in terms of comorbidities, overall health, and cancer pathology. Previous studies have indicated that eligibility criteria concepts may be disease specific and differences in patterns of concepts for eligibility criteria were found between breast cancer and cancer in general13. As a result, eligible patients must meet complex criteria, which are not standardized and vary by disease. In addition, not all cancer patients react similarly to the same therapeutic treatment. Heterogeneity in patient response to cancer therapies likely is caused by inter-individual differences in drug deposition and pharmacokinetics14. As we enter the era of precision medicine, patients will not be assigned to drug trials without attempting to predict how they will respond to the therapeutic regimen15. This complex problem will require efficient information systems that will need to be built within the clinic workflow16. Major eligibility criteria offer one effective strategy for streamlining clinical trial recruitment in cancer centers and to potentially increase patient satisfaction by limiting wait-times and maximizing care quality by quickly presenting all available treatment options to patients during clinical encounters.
4. Our previous pilot study
We previously conducted a pilot study involving nine clinical researchers, including physicians, nurses, and clinical research coordinators. Participants were invited to take a survey about how they use eligibility criteria to screen patients for clinical trials. Our study revealed a concept known as “major eligibility criteria”, which could be useful to assist investigators by automatically prescreening patients with data kept in the electronic health record. However, the responses revealed no clear consensus for the definition of “major eligibility criteria”. Each expert panelist defined the concept slightly differently, yet we identified frequently cited major criteria, such as histology and disease stage of cancer, in the aggregate data. Trends also emerged regarding how the panelists used major eligibility criteria in practice. For example, panelists described how they use major eligibility criteria to prescreen patients. The results also uncovered disparities about which criteria should be considered major. Our pilot study helped us to identify the unanswered research question and informed this study, which is to perform in-depth interviews and a focus group to understand the design requirements for prescreening with major eligibility criteria. The pilot study also suggested that major eligibility criteria can potentially improve triage efficiency for clinical trial recruitment by identifying patients that are potentially eligible for clinical trials. However, to the best of our knowledge, no prior study has investigated major eligibility criteria for clinical research recruitment and discussed user requirements for this type of decision support. This study employs a mixed-methods approach that includes both qualitative and quantitative measures to understand how clinical researchers define, select, and use major eligibility criteria1. Moreover, we harness their collective knowledge to inform future informatics designs for triaging patients for clinical research eligibility prescreening.
Methods
1. Study setting
We recruited physicians, nurses, and study coordinators from the Breast Cancer Program in the Herbert Irving Comprehensive Cancer Center (HICCC) at Columbia University Medical Center (CUMC). CUMC is an academic medical center in New York City that is affiliated with the New York-Presbyterian Hospital. Clinical researchers in the Breast Cancer Program at HICCC were sent an IRB-approved recruitment email with details about the study and invited to request more information. The protocol activities consisted of one initial interview that was audio recorded, one focus group session, and one survey after the focus group session that study participants completed independently. The invitees were informed that completing the survey was voluntary and they could opt out if they did not wish to participate. Eleven people requested information and were subsequently consented to the research study. We recruited eleven clinical research professionals, including five physicians, two nurse practitioners, one nurse, and three research coordinators. Two participants were male and nine were female. Their work experience with breast cancer clinical trials was an average of 6.8 years with a median of 6 years. All participants had experience in recruiting patients for breast cancer clinical trials. Figure 1 graphically depicts our study schema.
Figure 1.
The study schema.
2. The eligibility criteria dataset
Thirty clinical trials were randomly selected from the portfolio of active interventional breast cancer clinical trials, which represented over 41% of the 72 available cancer trials at the point of study at CUMC. Our random sample included a mixture of protocol phases to create a representative sample. All of our selected trials purposely involve interventions done to the patient. Twenty trials were designated as the training dataset, while the remaining ten were the testing dataset. Since significant differences exist in the content and word counts between full text protocols and their summaries on ClinicalTrials.gov3, which casts doubt on the reliability of clinical trial summaries, we decided to use eligibility criteria from full-text clinical trial protocol documents for this study.
3. The initial interviews
Ten research members participated in the initial interview to review eligibility criteria for six clinical trials and classified each criterion as either major or minor, so that each trial had three raters. Participants were given the title and all eligibility criteria for each clinical trial that they rated. The response rate was 100%. The survey was conducted using Qualtrics, an online survey platform (www.qualtrics.com). Participants were encouraged to “think aloud” while they completed the task so we could capture their thought process with audio recording, which was later transcribed. All identifying information was stripped from the audio recordings to ensure anonymity. The answers were downloaded from the Qualtrics website into Excel for analysis. The answers for the major eligibility criteria were analyzed individually for themes and then annotated by author MP, who is a subject matter expert in breast cancer clinical trial data collection with over nine years of experience. The initial interview completion rate was 100%. We assigned raters using a cascading structure to maximize variability in rater assignments, which ensured that each rater works with different participants on each clinical trial to maximize result validity.
4. The focus group
The focus group was held one week after completing the initial interviews. Eight participants, including five physicians, a nurse practitioner, a nurse, and a study coordinator, participated in the focus group. The focus group method was used to find areas of agreement and disagreement and participants were encouraged to have a discussion among themselves to complete the tasks17–18. The participation rate of the focus group was 100%. The group started the session by drawing their perception of how the current pre-screening process works. The participants worked individually and were instructed to focus on how physicians, nurses, and study coordinators interact with one another during the prescreening process, if at all. Participants were also asked how, when, and where prescreening occurs.
Next, the group reviewed eligibility criteria that were considered major from the initial interviews using two steps. First, the results of selected trials were reviewed with all focus group participants at the same time. During the discussion, Author MP listed the qualities and attributes of major eligibility criteria on an easel pad to learn why participants considered some eligibility criteria to be major. Author MP updated the list as the conversation progressed during the focus group session. Second, the focus group reviewed their discussion and agreed upon a list of attributes for major eligibility criteria based upon their subject matter expertise and the discussion of the focus group. This data was recorded using an easel pad that was visible throughout the focus group session, where anyone could write a quality or attribute about major eligibility criteria during the conversation. Then, the focus group reviewed the attributes and came to a consensus on a final list of attributes. Finally, the focus group participants discussed and drew their version of an optimized prescreening scenario as a group using easel pads that were hung on the wall so all focus group members could participate.
5. The follow-up survey
The follow-up survey was designed to evaluate the standardized major eligibility criteria and desiderata that were derived from our content analysis of the initial interviews and focus group session. Our content analysis was completed in two steps. First, we extracted all eligibility criteria from the training data set of twenty breast cancer clinical trials that were considered major by all three raters and referred to them as consensus major eligibility criteria. Second, we reviewed the list of attributes created by the focus group for major eligibility criteria. To ensure saturation of major eligibility criteria attributes, both authors reviewed and annotated the transcripts of the initial interviews until saturation was reached. Both authors reviewed the interview transcripts separately and discussed their results before coming to a consensus on major eligibility criteria attributes. Differences were resolved by discussion among authors. The result was the desiderata for major eligibility criteria in breast cancer clinical trials.
Ten participants completed the follow-up survey two weeks after the focus group session. The follow-up survey was administered using the Qualtrics survey tool and consisted of three parts. First, study participants were asked to classify each standardized major eligibility criterion that was created from the initial interviews using a 5-point Likert scale with options “Definitely Major”, “Possibly Major-Depending on the Context”, “Undecided”, “Possibly Minor”, and “Definitely Minor”19. The second part of the follow-up survey asked study participants to match examples of standardized eligibility criteria with each attribute of our derived major eligibility criteria desiderata. From the training dataset, twelve major eligibility criteria appeared in the training dataset more than once, so we included them in follow-up survey to have study participants match them with the derived desiderata. Study participants were directed to select as many or as few examples of major eligibility criteria for each attribute in our desiderata. The final part of the follow-up survey asked study participants to classify eligibility criteria as major or minor. Ten breast cancer clinical trials were randomly assigned to our ten participants so each participant would classify the eligibility criteria as major or minor for three testing clinical trials. We annotated the major eligibility criteria collected in the follow-up survey to see if any additional standardized major eligibility criteria were detected.
6. Data collection and analysis
Study data was collected with three methods: an online survey tool, Qualtrics, where the results were downloaded into Excel; audio recording and subsequent transcription; and the focus group session that created individual and group prescreening workflow artifacts and easel pad drawings. We utilized the “Think-aloud” Protocol to record the thought process of the study participants while they classified the training set eligibility criteria as either major or minor in the initial interview20. Then, we analyzed and annotated the audio transcriptions for the initial interviews using grounded theory to ensure saturation of the desiderata for major eligibility criteria from the focus group easel pads21. Finally, we asked study participants to evaluate the standardized major eligibility criteria as well as the desiderata in the follow-up survey.
Results
1. Qualitative study results
The average, median, maximum, and minimum time spent on the interview per participant was 22 minutes 42 seconds, 19 minutes 48 seconds, 40 minutes and 1 second, and 9 minutes and 42 seconds, respectively. Our interview transcript annotation revealed that all study participants differentiated major and minor eligibility criteria. There was 100% agreement amongst the participants that not all eligibility criteria should be given equal weight in the prescreening process. Participant 07 stated, “In my mind major eligibility criteria, I mean like who’s actually worth approaching upfront”. All study participants were able to find one or more major eligibility criteria for every clinical trial that they rated. Several major eligibility criteria were noteworthy and were reported below.
Disease staging was considered a major eligibility criterion for 100% of the study participants. Disease staging appears to be the most important factor to divide patients into subgroups for treatment decisions and recruitment for clinical trials. Participant 03 stated, “So, the disease staging, I would consider major. That’s your number one.” Participant 05 stated, “staging, this is probably one of the most major, most important”. This finding was confirmed by both transcription annotation and the follow-up survey, where 100% of participants identified disease stage as “Definitely Major”.
Transcript annotation also revealed a contextual sentiment about age. Several study participants stated that age could be considered a major or minor eligibility criterion depending on the context, such as how the numerical value of the age creates subgroups in the overall breast cancer population. For example, the age criterion can be major when its threshold is meaningful, such as “greater than 50 years old”. In contrast, the age criterion “adults greater than 18 years old” is not major because the majority of the patient population satisfies this criterion. Participant 04 stated, “I’m assuming that there’s going to be (over 18)”. Participant 09 stated, “…most of the patients that we see are over 18, so that’s usually an assumption I make that I would call minor, definitely”. The reasoning for this is the degree to which the age value divides the overall breast cancer population into meaningful subgroups. Participant 03 stated, “Most people at least in this setting are going to be 18. When the cut off is larger, like 50, it’s when I would consider it more of a major (criterion)”.
Physician, nurse, and study coordinator participants did not consider rare phenomena to be major eligibility criteria. Regarding previous cancers, participant 06 stated, “…it wouldn’t be considered a major criterion because for most patients, that’s not a major issue”. Participant 01 speaking about Hepatitis and HIV stated, “While that’s important, it’s not common”. Participant 03 stated, “Bilateral malignancy is relatively uncommon, I would consider that minor”. Participant 09 stated, “I would call that minor because I would say that the majority of the population is not HIV-positive”. Participant 02 stated, “The things that are less common become in my mind not major”.
An area of contention in the transcript annotation was laboratory results. Specifically, a difference existed between a study coordinator and nurses. Participant 08, a study coordinator, stated, “I can assess the labs…so that’s a major to me”. Participant 09, a nurse practitioner, stated, “I would say in terms of pre-screening, labs are minor because the lab is usually the one we use when they actually come in for their screening visit”. Participant 07, a nurse practitioner, stated, “let’s say the patient is essentially eligible except for some lab variations, for the most part in my experience, this can be remedied”. However, the importance of laboratory results may increase for patients with advanced disease. Participant 07 stated “I feel as though the laboratory parameters…have more implications for patients who are treated in a metastatic setting because those women tend to be more sick”.
A major eligibility criterion that generated significant conversation in the focus group session was around the concept of “prior treatment”. While prior treatment was not always considered to be accessible in the electronic health record, the focus group did consider it a major eligibility criterion that was available. However, it might take additional time to receive records from other clinicians that treated the patient. Focus group participants clarified that a patient’s prior treatment can be difficult to find, especially if they have been treated at multiple institutions. The focus group stated that the importance of prior treatment differs between early stage and advanced stage breast cancer patients and hence whether prior treatment is major is also a contextual decision.
The disease staging of the patient may increase the likelihood of other eligibility criteria being considered major. For example, the focus group agreed that the importance of prior treatment proportionally increases in connection with the number of patient prior treatments. For a patient with multiple prior treatments, it is likely that prior treatment becomes a major eligibility criterion for that specific patient. Similarly, if a patient has not had any prior treatments, the concept of prior treatment is less important for that individual patient. This sentiment was also reflected in the transcription annotation. Participant 02 stated during the initial interview, “Given that this is a neoadjuvant study, most patients would not have had prior therapy. So probably not so important to know that they haven’t had prior treatment”. Therefore, prior treatment is more relevant to patients with advanced disease than for early stage disease.
2. Desiderata for major eligibility criteria
Desiderata for major eligibility criteria (Table 1) were derived based upon the transcript annotation from the interviews using grounded theory as well as from a group activity that was conducted during the focus group session. The desiderata were then evaluated in the follow-up survey. The desiderata attributes are ranked in descending order of frequency determined by counting the matched eligibility criteria for each term in the desiderata for all participants that completed the follow-up survey. The twelve most frequent major eligibility criteria from the training data set were used for the major eligibility criteria examples. In addition, the major eligibility criterion with the highest score from study participants is identified as a “Best Criterion Example”, where we have included more than one criterion in the event of a tie.
Table 1.
Desiderata of major eligibility criteria for breast cancer clinical trials in descending order of frequency.
| Desiderata | Operational Definition | Rationale Quotes from Study Participants | Best Criterion Example in the Follow-up Survey |
|---|---|---|---|
| Reliable | A good biomarker | “I guess the question is how accurate are the data that are going to be in the EMR.” | Disease staging, HER2 status; Hormone receptor status |
| Objective | Assessed in a standardized way without opinions | “one way for me to delineate majors versus minors, …something that’s like blatantly objective, it’s major.” | Age |
| Relevant to me | Something I can review on my own | “that’s going to be pathology review, and I don’t have experience on that” | Eligible to receive treatment |
| Available | Information ready to use | “there’s always things that aren’t going to be in the medical record or things that you’re not going to know.” | Disease staging |
| Stable | No fluctuation in the short term | “If there is something that the patients can discontinue prior to randomization, then that would be minor.” | Diagnosis made by core biopsy; Gender |
| Intrinsic patient trait | Something that cannot be changed by external force | “let’s say the patient is essentially eligible except for some lab variations, for the most part in my experience, this can be remedied.” | Age; Gender |
| Accessible | Understandable with little effort | “things that would be easy to prescreen” | Age |
| Relevant to now | Something I can find out now | “that won’t happen until we actually meet them.” | Age; Gender |
| Prevalent in population | Concepts common to breast cancer patients | “wouldn’t be considered a major criterion because for most [breast cancer] patients that’s not a major issue” | Disease staging |
| Must have | Not optional | “Must be post-menopausal, major.” | Disease staging |
| Differential for population subgroups | Able to divide population into subgroups | “…age > 18 is not a major criterion but specific age constraints such as between 50 and 75 are major criteria” | Disease staging |
| Plausible (to happen) | Something realistic and practical | “We would never approach a person with life threatening metastasis” | Eligible to receive treatment; Scheduled to receive treatment |
3. The prescreening process
During the initial interviews, participants identified different phases of patient screening. Participant 06 stated that major eligibility criteria are used for prescreening, while minor criteria will be used during manual reviews of data to give higher scrutiny to the patient. An example of a minor criterion is asking a patient if they would consider participating in a research trial. The transcript annotation revealed that different phases of patient screening occur at different time points. Participant 06 identified three different steps to screening. The first step is looking at “high level variables”. The second step is when you give a patient a more in-depth look for obvious problems that would prevent them from enrolling in the clinical trial such as organ failure. Thirdly, you have the official eligibility checklist, where you go line by line and see if they meet all eligibility criteria.
At the beginning of the focus group session, study participants were asked to draw the workflow of how patients are currently prescreened in their clinics. 100% of the focus group participants completed the task. Seven out of eight participants identified the first step of the prescreening process starting with the physician, where the physician directs nurses or study coordinators to do a manual screening of the patient. The one remaining participant stated that the prescreening could start with any clinical researcher, depending on the type of study. For example, study coordinators may recruit participants for behavioral or observational studies without physician input.
The final group activity of the focus group was to describe an optimized prescreening workflow. The prescreening process utilizes major eligibility criteria to automatically identify potentially eligible patients in the electronic health record. Study participants came to a consensus that decision support for clinical trials must be built into the everyday workflow of the clinicians. At CUMC, the clinicians view a patient schedule that is built for each physician’s clinic day. The focus group agreed that research staff should then be able to easily see potentially eligible patients in the physician’s daily schedule. This enables research staff to screen patients and communicate with one another to make an ultimate screening decision.
Figure 2 shows the optimized prescreening workflow as developed by consensus during the focus group session. In addition to describing the workflow, focus group participants also illustrated the type of information that would be useful in the user interface (Yes; No; Maybe). The relevant clinical trials for each patient are listed on the physician schedule. For example, Patient 1 is scheduled to see the physician at 8:00 AM and is potentially eligible for “Trial A” and “Trial B” as seen in Figure 2. The automated prescreening process, study coordinator, research nurse, and physician can each select “No–Not Eligible, “Yes–Possibly Eligible”, and “Maybe Eligible” after their respective reviews.
Figure 2.
Optimized prescreening workflow.
4. Quantitative study results
The quantitative results include the instance counts from the training and testing datasets as well as the Likert scale measurements from the follow-up survey. For the training dataset that composed twenty breast cancer trials, the average count of eligibility criteria per trial was 19.7 with a median of 18. There was more than double the amount of criteria that were classified as minor than were classified major. The average major criterion per clinical trial in the training set was 3.55. The average minor criterion per clinical trial in the training data set was 7.3. Similarly, the testing dataset had an average count of 22.5 total eligibility criteria with a median of 20. We again saw there was more than double the amount of criteria that were classified as minor than were classified major. The average major criterion per clinical trial in the training set was 3.2. The average minor criterion per clinical trial in the training data set was 6.9. We annotated the study participant agreed upon major eligibility criteria from the initial interviews as well as from the follow-up surveys. We did not detect any new standardized major eligibility criteria in the follow-up survey. The instance count of the training and testing datasets are seen in Table 2 sorted by total dataset instances. Study participants completed a 5-point Likert scale to classify each major eligibility criterion as “Definitely Major”; “Possibly Major–Depending on the Context”; “Undecided”; “Possibly Minor”; and “Definitely Minor”. We report the mode from participant responses from the follow-up survey Likert scale in Table 222.
Table 2.
Standardized major eligibility criteria with instance counts and Likert scale mode.
| Major Eligibility Criterion | Training Instances | Testing Instances | Total Dataset Instances | Likert Scale Mode for the Participant Group |
|---|---|---|---|---|
| Disease staging | 20 | 12 | 32 | Definitely Major |
| Prior treatment | 17 | 8 | 25 | Definitely Major |
| Hormone receptor status | 9 | 3 | 12 | Definitely Major |
| Gender | 7 | 2 | 9 | Possibly Major |
| Histological confirmed diagnosis | 6 | 2 | 8 | Definitely Major |
| HER2 status | 5 | 3 | 8 | Definitely Major |
| Pre/Postmenopausal status | 5 | 2 | 7 | Possibly Major |
| Comorbidities | 3 | 3 | 6 | Possibly Major |
| Eligible to receive treatment | 2 | 2 | 4 | Definitely Major |
| Diagnosis made by core needle biopsy | 4 | 0 | 4 | Definitely Major |
| Age | 2 | 1 | 3 | Possibly Major |
| Laboratory values | 1 | 2 | 3 | Possibly Major |
| At least one breast available | 1 | 1 | 2 | Possibly Major |
| Scheduled to receive treatment | 2 | 0 | 2 | Possibly Minor |
| Measurable disease | 1 | 0 | 1 | Definitely Major |
| Future scheduled treatment | 1 | 0 | 1 | Possibly Minor |
| Blood pressure | 1 | 0 | 1 | Possibly Major |
| Language | 1 | 0 | 1 | Undecided |
| Heart rate | 1 | 0 | 1 | Possibly Major |
| Family history of breast cancer | 1 | 0 | 1 | Possibly Minor |
| Ethnicity | 1 | 0 | 1 | Possibly Minor |
| Suitable to undergo MRI | 1 | 0 | 1 | Possibly Minor |
| Have mobile phone | 1 | 0 | 1 | Definitely Minor |
Figure 3 below displays major eligibility criteria as ranked by study participants in the final survey in decreasing order of significance23. The criteria are ranked by significance score, which was calculated by summing all participant scores for each criterion. The criteria received the following scores for each classification: “Definitely Major”=1; “Possibly Major–Depending on the Context”=2; “Undecided”=3; “Possibly Minor”=4; and “Definitely Minor”=5. The criterion with the lowest score is considered the most significant. For example, “Disease staging” had the lowest significance score because all participants identified it as “Definitely Major”.
Figure 3.
Major eligibility criteria measured with a Likert scale by participants in the follow-up surveys.
Discussion
Our training dataset was considered a representative sample of interventional breast cancer clinical trials at CUMC for three reasons. First, our results showed that “Hormone receptor status” was identified as the third most common major eligibility criterion with 12 total instances in our dataset and second most significant major eligibility criterion after disease staging. This observation confirmed sample representativeness since the prevalence of hormone receptor positive breast cancer in the entire breast cancer population is approximately two-thirds of all reported cases24. Second, saturation was reached in our training dataset of major eligibility criteria instances since no new major eligibility criteria were detected in our testing dataset. Third, the sample represented over 41% of interventional breast cancer trials available at CUMC.
The results of our mixed-methods study prove the existence of major eligibility criteria. In addition, both the training and testing datasets had similar summary statistics. The average count of total eligibility criteria for both training and testing datasets was 19.7 and 22.5, respectively. The average count of major eligibility criteria for training and testing data sets was 3.55 and 3.2, respectively. Minor eligibility criteria are more prevalent in our sample of clinical trials than major criteria. This finding is significant because automating a system that needs to search for fewer machine-readable standardized criteria is more efficient and feasibly operational. In addition, this finding questions prior efforts to provide a complex knowledge representation of highly expressive concepts that gives equal weight to all eligibility criteria. Our study suggests that time could be saved by focusing on major criteria that were identified by study participants. It is possible that raters identify minor criteria more easily than major criteria, but an inescapable conclusion is that complex knowledge representations that give equal weight to all criteria are perhaps less useful than previously thought. This is an area that requires further exploration.
We have derived a standardized list of major eligibility criteria for breast cancer clinical trials at CUMC. We have confidence in our list being complete because we did not detect any new standardized major eligibility criteria in our testing dataset during the follow-up survey. While we focused exclusively on breast cancer, our study provides a mixed-methods approach for discovering major eligibility concepts in other diseases. This approach will be helpful in future projects since clinical trials lack standardized eligibility criteria. As a consequence, recognizing similar criteria across two or more clinical trials remains a challenge even in the same disease realm.
Our approach differs from previous research attempts to create eligibility criteria knowledge representations by avoiding highly expressive standardization and temporal restraints in favor of focusing on only the most user-relevant concepts. The granularity of the major eligibility criteria desiderata is coarser than other knowledge representations and more comparable to available electronic health record data. The coarseness of our standardized criteria is a very strong advantage for the usefulness of major eligibility criteria because it takes into account practical considerations such as data availability in electronic health records and fitness to the clinic workflow. Our focus group detailed how clinical research professionals use major eligibility criteria in their thought process while prescreening patients. Focus group data provide insight into building a clinical decision support system that is powered by major eligibility criteria to automatically suggest clinical trials that patients are potentially eligible for during clinical encounters.
We also sought to understand the rationale of major eligibility criteria. The study participants viewed HIV as less important for breast cancer clinical trial recruitment because it is rare among the target population. Other criteria, such as age, depend on context. Annotated transcripts from the initial interviews, the focus group session, and the follow-up survey provide evidence that prior treatment is significant, yet difficult to obtain. Thus, prior treatment is difficult to utilize in automatic prescreening. More advanced disease staging may increase the importance of some eligibility criteria such as laboratory results and prior treatment. As a consequence, clinical trials for early stage breast cancer could realize more of a benefit using major eligibility criteria for automated prescreening than studies for advanced disease stages.
The results of our mixed-methods study also identify the desiderata of major eligibility criteria for breast cancer clinical trials. While we utilized only breast cancer clinical trials, our desiderata could be scalable to other diseases. Furthermore, the desiderata could provide guidance about what features will create an optimal clinical trials recruitment tool for future clinical decision support systems. Since the desiderata terms are ranked by frequency, they provide weighted insight into how clinical researchers think while prescreening patients for clinical trials. For example, “Reliable” was the desiderata attribute that was most frequent to our major eligibility criteria followed by “Objective” and “Relevant to me”. The relatively highly ranked “Relevant to me” desiderata attribute suggests that a role-based analysis may provide additional insight into prescreening processes.
Limitations
Since the results are only from one academic medical center, we cannot speculate to the universality of the desiderata. We purposely included only breast cancer clinical trials in our dataset because previous research findings suggest that relevant eligibility criteria are disease specific. The applicability of our findings to other diseases is unknown. This study did not use any real patient data, instead relying on observing the opinions and thought processes of clinical research professionals using the “Think-aloud” protocol. As a consequence, we are unsure what effect a fragmented or incomplete patient electronic health record would have on influencing the major eligibility criteria desiderata. Finally, even though our study methods included qualitative and quantitative methods, our small sample size prohibits us from making definitive conclusions about breast cancer clinical trials in a larger context.
Future work
The scalability of the major eligibility criteria desiderata to other diseases and institutions is an unanswered research question that warrants additional exploration. Our desiderata may assist clinical researchers and informaticists to make informed decisions about prioritizing the standardization and collection of data in the electronic health record for clinical trial prescreening. 100% of study participants agreed disease stage was “Definitely Major” during the follow-up survey. This suggests that we should prioritize the capture of this information in a standardized, machine-readable way for efficient prescreening. Similarly, “Hormone receptor status” and “HER2 status” should be prioritized for standardized capture. Standardized major eligibility criteria could provide a basis for clinical decision support by triaging patients for clinical trial prescreening and other applications in personalized medicine.
Future work should evaluate how major eligibility criteria are captured in electronic health records and how prescreening results can be best presented to clinical researchers in the clinic workflow. Our study has shown how major eligibility criteria might be used in an optimized workflow to screen patients for clinical trials. Automated prescreening could reduce human effort and expense by reducing the manual review of patient charts. If financial or other barriers prevent major eligibility criteria from being collected in a standardized way or even preclude an automated prescreening process, the benefits of major eligibility criteria still exist. For example, major eligibility criteria can still inform the development of manual prescreening methods. Additional evaluation of the desiderata and standardized major eligibility criteria will be helpful for understanding how our research can be utilized when prescreening patients in real life scenarios.
Conclusion
Our results demonstrate that major eligibility criteria exist for breast cancer clinical trials. This study investigated how, when, and why clinical researchers use major eligibility criteria to triage patients for clinical trial prescreening. Our identified desiderata can potentially inform future design of clinical decision support for clinical trial prescreening. Future research is warranted to test the generalizability of our method beyond breast cancer trials and consider real patient data in order to gain insight into how the major eligibility criteria desiderata is influenced by fragmented or incomplete patient records and what steps might be taken to mitigate those effects.
Acknowledgments
This study was sponsored by the U.S. National Library of Medicine grant R01LM009886 (PI: Weng) and U.S. National Center for Advancing Translational Science grant UL1 TR000040 (PI: Ginsberg).
References
- 1.Maxwell JA. Using numbers in qualitative research. Qualitative Inquiry. 2010;16:475–82. [Google Scholar]
- 2.Weng C, Batres C, Borda T, Weiskopf NG, Wilcox AB, Bigger JT, et al. A real-time screening alert improves patient recruitment efficiency. AMIA Annual Symposium Proceedings. 2011;2011:1489. [PMC free article] [PubMed] [Google Scholar]
- 3.Bhattacharya S, Cantor MN. Analysis of eligibility criteria representation in industry-standard clinical trial protocols. Journal of Biomedical Informatics. 2013;46(5):805–13. doi: 10.1016/j.jbi.2013.06.001. [DOI] [PubMed] [Google Scholar]
- 4.Sullivan J. Subject recruitment and retention: barriers to success. Applied Clinical Trials. 2004 [Google Scholar]
- 5.Rubin RR, Fujimoto WY, Marrero DG, Brenneman T, Charleston JB, Edelstein SL, et al. The diabetes prevention program: recruitment methods and results. Controlled clinical trials. 2002;23(2):157–71. doi: 10.1016/s0197-2456(01)00184-2. [DOI] [PubMed] [Google Scholar]
- 6.Tu SW, Kemper CA, Lane NM, Carlson RW, Musen MA. A methodology for determining patients’ eligibility for clinical trials. Methods Inf Med. 1993;32(4):317–25. [PubMed] [Google Scholar]
- 7.Damen D, Van den Bulcke T. Towards a flexible semantic framework for clinical trial eligibility using topic maps; Proceedings of the ACM SIGKDD Workshop on Health Informatics, HI-KDD; 2012. [Google Scholar]
- 8.Tu SW, Peleg M, Carini S, Bobak M, Ross J, Rubin D, et al. A practical method for transforming free-text eligibility criteria into computable criteria. J Biomed Inform. 2011;44(2):239–50. doi: 10.1016/j.jbi.2010.09.007. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Weng C, Wu X, Luo Z, Boland MR, Theodoratos D, Johnson SB. EliXR: an approach to eligibility criteria extraction and representation. Journal of the American Medical Informatics Association. 2011;18(Suppl1):i116–i24. doi: 10.1136/amiajnl-2011-000321. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Boland MR, Tu SW, Carini S, Sim I, Weng C. EliXR-TIME: a temporal knowledge representation for clinical research eligibility criteria. AMIA Summits Transl Sci Proc. 2012;2012:71–80. [PMC free article] [PubMed] [Google Scholar]
- 11.Weng C, Tu SW, Sim I, Richesson R. Formal representation of eligibility criteria: a literature review. J Biomed Inform. 2010;43(3):451–67. doi: 10.1016/j.jbi.2009.12.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Gentles AJ, Gallahan D. Systems biology: confronting the complexity of cancer. Cancer Research. 2011;71:5961–4. doi: 10.1158/0008-5472.CAN-11-1569. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Milian K, Bucur A, van Harmelen F, ten Teije A. Identifying most relevant concepts to describe clinical trial eligibility criteria. Health Inf. 2013 [Google Scholar]
- 14.van der Greef J, Hankemeier T, McBurney RN. Metabolomics-based systems biology and personalized medicine: moving towards n= 1 clinical trials? Pharmacogenomics. 2006;7(7):1087–94. doi: 10.2217/14622416.7.7.1087. [DOI] [PubMed] [Google Scholar]
- 15.West M, Ginsburg GS, Huang AT, Nevins JR. Embracing the complexity of genomic data for personalized medicine. Genome research. 2006;16(5):559–66. doi: 10.1101/gr.3851306. [DOI] [PubMed] [Google Scholar]
- 16.Tu SW, Musen MA. Modeling data and knowledge in the EON guideline architecture. Stud Health Technol Inform. 2001;84(Pt 1):280–4. [PubMed] [Google Scholar]
- 17.Kitzinger J. Qualitative Research: Introducing focus groups. BMJ. 1995:299–302. doi: 10.1136/bmj.311.7000.299. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Kidd PS, Parshall MB. Getting the focus and the group: enhancing analytical rigor in focus group research. Qualitative Health Research. 2000;10:293–308. doi: 10.1177/104973200129118453. [DOI] [PubMed] [Google Scholar]
- 19.Likert R. A technique for the measurement of attitudes. Archives of psychology. 1932 [Google Scholar]
- 20.Kushniruk AW, Patel VL, Cimino JJ. Usability testing in medical informatics: cognitive approaches to evaluation of information systems and user interfaces. Proceedings of the AMIA Annual Fall Symposium. 1997:218–22. [PMC free article] [PubMed] [Google Scholar]
- 21.Martin PY, Turner BA. Grounded theory and organizational research. The Journal of Applied Behavioral Science. 1986;22(2):141–57. [Google Scholar]
- 22.Boone HN, Boone DA. Analyzing likert data. Journal of Extension. 2012;50(2):1–5. [Google Scholar]
- 23.Robbins NB, Heiberger RM, editors. Plotting likert and other rating scales; Proceedings of the 2011 Joint Statistical Meeting; 2011. [Google Scholar]
- 24.Traina TA, Rugo HS, Caravelli JF, Patil S, Yeh B, Melisko ME, et al. Feasibility trial of letrozole in combination with bevacizumab in patients with metastatic breast cancer. Journal of Clinical Oncology. 2010;28(4):628–33. doi: 10.1200/JCO.2009.21.8784. [DOI] [PMC free article] [PubMed] [Google Scholar]