Skip to main content
PMC home page
Applied Clinical Informatics logoLink to Applied Clinical Informatics
. 2022 Mar 30;13(2):363–369. doi: 10.1055/s-0042-1743560

Searching of Clinical Trials Made Easier in cBioPortal Using Patients' Genetic and Clinical Profiles

Philipp Unberath 1,✉, Lukas Mahlmeister 1, Niklas Reimer 2, Hauke Busch 2, Melanie Boerries 3,4, Jan Christoph 1,5
PMCID: PMC8967483  PMID: 35354211

Abstract

Background  Molecular tumor boards (MTBs) cope with the complexity of an increased usage of genome sequencing data in cancer treatment. As for most of these patients, guideline-based therapy options are exhausted, finding matching clinical trials is crucial. This search process is often performed manually and therefore time consuming and complex due to the heterogeneous and challenging dataset.

Objectives  In this study, a prototype for a search tool was developed to demonstrate how cBioPortal as a clinical and genomic patient data source can be integrated with ClinicalTrials.gov, a database of clinical studies to simplify the search for trials based on genetic and clinical data of a patient. The design of this tool should rest on the specific needs of MTB participants and the architecture of the integration should be as lightweight as possible and should not require manual curation of trial data in advance with the goal of quickly and easily finding a matching study.

Methods  Based on a requirements analysis, interviewing MTB experts, a prototype was developed. It was further refined using a user-centered development process with multiple feedback loops. Finally, the usability of the application was evaluated with user interviews including the thinking-aloud protocol and the system usability scale (SUS) questionnaire.

Results  The integration of ClinicalTrials.gov in cBioPortal is achieved by a new tab in the patient view where the genomic profile for the search is prefilled and additional parameters can be adjusted. These parameters are then used to query the application programming interface (API) of ClinicalTrials.gov. The returned search results subsequently are ranked and presented to the user. The evaluation of the application resulted in an SUS score of 83.5.

Conclusion  This work demonstrates the integration of cBioPortal with ClinicalTrials.gov to use clinical and genomic patient data to search for appropriate trials within an MTB.

Keywords: clinical decision support, genomics, clinical trial, molecular tumor board, trial matching, genetic alteration

Background and Significance

Genome sequencing has become more cost-effective and feasible due to technical developments such as next-generation sequencing (NGS) methods. 1 2 Oncology, in particular, benefits from these developments, as sequencing of the cancer genome can provide extended molecular tumor profiling, as well as valuable information on potential personalized treatment strategies. 3 This consideration has led to the development of a variety of targeted therapy approaches. 4 5 6

Since the same targetable mutations can occur in different tumor entities, more and more clinical trials are being conducted as so-called basket studies which include tumor entities based on its specific mutation or biomarker. 7 8 This leads to a much wider spread of tumor characterizations and therefore also to a reduction of matching patients and corresponding case numbers. 9

To cope with the complexity of molecular-based cancer treatment, many institutions have implemented molecular tumor boards (MTBs) complementing the already existing organ-based tumor boards. 10 These MTBs are composed of interdisciplinary experts reviewing and discussing the complex personalized therapy options based on clinical and advanced molecular diagnostics. For treatment recommendations, it is crucial to find accessible clinical trials that match the genetic profile of the patient's tumor to provide patient the opportunity to participate in these trials, or to include findings from completed trials. Especially in the last few years, it has been possible to expand the therapy guidelines based on genetic tumor profiling, as shown for example for lung carcinoma by Lindeman et al. 11 However, mainly due to the increase in advanced diagnostics using next generation sequencing, there are still other unknown genetic findings where no guided line therapy is currently available.

However, the search for suitable clinical trials is often performed manually, and therefore, it is a time-consuming process. 12 One tool that can be employed to use bioinformatic and clinical resources to determine the clinical relevance of gene alterations for the MTB is the open-source platform cBioPortal developed by the Memorial Sloan Kettering Cancer Center. 13 14 It allows analysis of clinical and molecular patient data of cancer patients and can be accessed as a public instance or deployed locally enabling customization of the portal and analysis of own patient data. Buechner et al performed an extensive requirements analysis and specification of an MTB platform 15 in which they found cBioPortal as a suitable basis for such an application. One of the identified still missing features for the use of cBioPortal in the clinical setting of an MTB was facilitating the search of suitable clinical studies for patients.

Utilizing cBioPortal as a local clinical and genomic data warehouse, there were already efforts to integrate clinical trials matching based on the genetic profile and clinical patient data. This is achieved by the integration of MatchMiner 16 that matches the patient's genetic profile to clinical trials that have previously been curated by domain experts and converted into a structured proprietary format. However, the curation constitutes a time-consuming process, therefore resulting in a relatively small number of available studies and a constant effort to keep the data up to date. Other solutions to find matching studies based on genetic characteristics are either proprietary, such as My Cancer Genome, 17 or single-institution, such as TrialProspector, 18 and therefore not publicly available.

Objectives

In this work, we developed an extension to cBioPortal to enable patient-centric clinical trials search. It is based on the integration of ClinicalTrials.gov and leverages cBioPortal's structured molecular and clinical data which can be used to prefill the search fields. The main requirement was that the search for clinical trials should not require any manual curation or preprocessing of trial data and should be executed on the fly from the fronted of cBioPortal. It should aid MTB participants during the often laborious process of searching clinical trials for MTB patients, and therefore support an important one of their routine tasks. To account for the specific needs and requirements of the MTB participants, a prototype was implemented using a user-centered design approach including multiple feedback loops and evaluated in the context of a usability test.

Methods

Buechner et al 15 revealed the need of the MTB for the previously described extension in their requirements analysis and also already created a high-fidelity mockup for the design thereof. Adding on their findings, an in-depth requirements analysis for the specifics of the extension using open expert interviews was conducted. The interviews for this requirements analysis were performed in the course of a biweekly jour fixe about the advancement of processes and software in MTBs with over 20 regularly participating clinicians and medical informatics specialists of nine German partner sites. Based on the mockup ( Fig. 1 ) and the resulting detailed requirements, a prototype was implemented using a user-driven development process with a total of three iterations over a period of 2 months. During this process, the prototype was further refined iteratively, taking into account the feedback of the real-world users during the aforementioned jour fixes, particularly oncologists who provide MTB therapy recommendations for patients. This feedback of the prototype was collected by allowing users to try out the prototype and report any problems, ambiguities, or missing features during the jour fixes or afterward as written feedback. The prototype used real patient data from the cBioPortal public database and real study data from ClinicalTrials.gov.

Fig. 1.

Fig. 1

Open in a new tab

The high-fidelity mockup of the frontend integration of the ClinicalTrials.gov search as proposed by Buechner et al. 15 It is derived from a screenshot of the patient view, adding the studies search tab using an image editing for the purpose of better demonstrating the feature. The mockup does not provide any functionality, but it served as a blueprint for the prototype.

To evaluate the extension, a formative usability test (the summative evaluation will be performed in a future study along with multiple other features extending cBioPortal for its use in MTBs) was conducted with real-world users. The call to participate in the usability test went to all potential probands, that is, 10 clinicians regularly attending MTBs and the aforementioned jour fixes. Five of these participants of the MTB were asked to perform a clinical trial search for a patient 19 using the newly implemented extension as the scenario for the usability test. During the test process, the participants were instructed to review the patient's genetic alterations and clinical data and select fitting data to feed into the search tool. In the first step of the search, the users were asked to prioritize studies the patient could realistically enroll in, most importantly meaning studies close to the patient's place of residence. The second step should also include worldwide studies from which the user may only derive new therapy approaches, as enrollment of own patients would not be practical due to the place of residence. During the whole process, the user was asked to verbalize all thoughts (think-aloud protocol). Afterward, the participant was interviewed using open-style and Likert's scale 20 questions and finally asked to complete the system usability scale (SUS) 21 questionnaire in German 22 which is the native language of all participants. The usability tests were performed as web meetings using the teleconferencing software Zoom, taking approximately 30 to 60 minutes, with screen sharing enabled which allowed the recording of the users' interactions with the user interface, as well as all feedback during the think-aloud protocol. These recordings were then systematically analyzed. All documented errors, difficulties, and hesitations from the usability perspective during the think-aloud protocol, as well as questions and suggestions for improvement from the open interview were grouped into categories and summarized to single change requests after collection.

Results

The detailed requirements analysis revealed that the search results should be displayed as a table within a new tab of the patient view of cBioPortal ( Fig. 2 ). The adjustable search parameters should consist of the following parameters: (1) variants and copy number variations (CNVs), (2) generic free-text keywords, (3) recruiting statuses of the study, (4) trial location by country, and (5) the age and sex of the patient. To be able to review the proposed trial results as fast as possible, the table should include (1) the recruiting status, (2) found keywords, (3) the title of the study, (4) conditions/diagnoses, (5) interventions, and (6) locations of partner sites. During the iterative feedback rounds with real-world users, prior requirements were adapted and refined and the prototype could be further developed. Multiple comments were related only to the user interface, for example, the way of entering study locations, how to select relevant alterations, or improving the explanation of fields using tooltips. But there were also suggestions requiring adaptions of the general requirements and therefore changes to multiple layers including the user interface and the logical handling of user input that affects the query of clinical trials. First, it should also be possible to differentiate between required and optional keywords and an option to specify whether they should be combined with “AND” or “OR”. Second, the location search should be improved to not only enable a search by country but a proximity search using a city as input.

Fig. 2.

Fig. 2

Open in a new tab

The prototype of the newly developed tab in the cBioPortal patient view of the ClinicalTrials.gov search.

The aforementioned search parameters are then used without further modification or enrichment to build a request to query the application programming interface (API) of ClinicalTrials.gov. 23 Scoring of the search results was achieved by first filtering the results (if specified) by country, recruiting status, and selected alterations. Afterward, the remaining trials were ranked by a combined weighted query based on feedback of the users during the iterative feedback rounds which consisted of (importance in descending order) count of found keywords, matching tumor entity, distance to closest partner site, matching sex, and matching age.

Next, different methods of integrating the extension were considered, including integration in the cBioPortal frontend (running in the user's browser), the cBioPortal backend (running on a dedicated server), and as a standalone service, running on the server in addition to the cBioPortal backend (for more information about the architecture of cBioPortal, refer to reference Unberath et al 24 ). Since cBioPortal does not provide a plugin concept, 24 the extension should be coupled as loosely as possible to the codebase to ensure better compatibility with future updates. This indicates an implementation in the frontend which also promotes the feature of retrieving search results on the fly. Since results are not cached, curated, or preprocessed in advance by the server, the display of results should be focused on ranking trials according to the number of matching criteria, listing results with less matches at the end rather than excluding them completely. This leaves the final assessment of the relevance of all found studies to the user. Another key feature of the extension stems from its usage in a clinical setting with real patients, the proximity search, which considers the patient's place of residence and ranks the studies with matching geographical sites higher accordingly.

Finally, the concluding usability evaluation was performed in October 2020 with five clinicians with different backgrounds (oncology, systems medicine, or bioinformatics), who regularly participate in MTBs. Several additional shortcomings were identified that were not identified during the feedback rounds and needed to be addressed for finalizing the tool. Most importantly, it was not clear to the users that by design the tumor entity was invariably part of the ranking of the tool in the background, as it is always present in cBioPortal samples. All users therefore demanded that ranking parameters should be made selectable and customizable for the query. Other important feedback included improving the explanation of how search results were filtered and ranked and how to enter the optional and required keywords (each mentioned by three participants). The participants also mentioned additional nice to have features, such as the ability to search for the phase of the study, for interventions or alterations based on pathways (each mentioned by one participant). The latter can be considered as a suggestion for improvement for further developments.

The results of the quantifiable questions of the usability evaluation can be found in Table 1 . The evaluation of the SUS questionnaire resulted in an overall SUS score of 83.5 (number of participants was five, standard deviation = 11.58, individual scores were 92.5, 95, 90, 65, and 75, respectively). The SUS score of 83.5 is in the middle of the “acceptable” range and represents a “good” on the adjective rating scale. 25

Table 1. The results of the quantifiable questions of the usability evaluation.

Question Mean SD
The function of each field of the search mask was comprehensible. 3.80 0.75
The information provided in the search results helped to identify suitable or unsuitable studies. 3.80 0.75
The order in which the studies are displayed was helpful. 3.25 1.01
The application overall facilitates the search for suitable studies. 4.00 0.71
Open in a new tab

Abbreviation: SD, standard deviation.

Note: Scores range from 1 (strongly disagree) to 5 (strongly agree).

Discussion

Clinical trials are a major driver in the advancement of cancer treatment and while the willingness of patients to participate in trials is estimated as high as 70%, 26 less than 5% of cancer patients enroll in clinical trials. 27 28 Unger et al 27 identified multiple barriers in different categories from which this work focuses on the structural barrier of identifying available clinical trials. Although being resource intensive, 29 the process of searching and identifying appropriate trials is a crucial part of the preparation of patients for the MTB, as these patients usually already have exhausted all guideline-based treatment options or have rare tumors. 30 31

While there are other works regarding software solutions for MTBs, these developments are still in early stages. Halfmann et al 32 developed an MTB support tool also using a user-driven development process. Their tool focuses on the presentation part of an MTB and not on the preparation including the search for clinical trials for MTB patients and furthermore, the study did not include a usability evaluation. Fegeler et al 33 are currently developing a solution focusing on the administrative part including communication methods for performing virtual tumor boards. Studies in the field of recruitment support often tackle different aspects than searching fitting trials for a specific patients, for example, optimize eligibility criteria 34 or assess trial population representativeness, 35 supporting data collection 36 or helping to find patients for screening. 37 38

A key factor in enrollment of MTB patients in clinical trials are matching genomic alterations of the patient's tumor and the study's inclusion criteria, yet these alterations are not specified in a well-structured matter and often only found study's descriptive text. 39 Therefore, several attempts have been made to curate precision cancer studies from trial registries like ClinicalTrials.gov or institution-specific registries to generate a structured database to query. The drawback of these approaches generally is the high maintenance of curating trials and keeping the data up to date. 40 Additionally, institution-specific systems, like MatchMiner 16 or the Phase One Spot Tracker (POST), 41 do not publish curated datasets. While those data would not directly benefit the enrollment at other institutions (depending on the proximity), they would constitute a valuable resource for benchmarking or validating the results of new matching tools. However, such a validation was not in the scope of this work and was therefore not performed. The focus was on streamlining the process of the manual search by integrating ClinicalTrials.gov into cBioPortal and prefilling data already available for the patient.

In contrast to aforementioned tools, this work focuses on finding studies by directly using the necessary information for the search from the patient data already available in cBioPortal. It is obvious that such an approach can never achieve sensitivity and specificity levels of tools that require upfront efforts from domain experts. Therefore, this tool focuses on a high usability to enable physicians to query up to date and publicly available trial registries (currently solely ClinicalTrials.gov) on the fly as fast and straightforward as possible. The tool does have technical limitations, some of which originating from the API of ClinicalTrials.gov. For instance, while the advanced search of the ClinicalTrials.gov web interface features a practical distance search, the API lacks this feature. Thus, the extension implements an own distance search using a set of predefined locations which needs to be further expanded. For the search of synonyms rather than exact string matches, the API provides the same functionality as the web interface, utilizing the Medical Subject Headings (MeSH) 42 to search for all synonyms of the given term. Currently, the extension uses the HUGO Gene Symbol from cBioPortal for querying the API, but in the context of molecular profiling, it could be beneficial to also include more specialized vocabularies, such as Gene Ontology, 43 for a broader synonym coverage, and using gene identifiers as HUGO symbols may change over time. 44 Future developments could also consider pathways of the patient's genes affected by mutations.

One approach to improve the fit of our tool's search results would have been to preprocess or automatically curate trials from ClinicalTrials.gov in advance. Previous studies shown that even sophisticated curation pipelines require manual posttreatment. 39 This fact also discouraged the implementation of an extract, transform, load (ETL)-pipeline from ClinicalTrials.gov to the MatchMiner trial format Clinical Trial Markup Language (CTML). Preprocessing trials in the form of building a custom registry would have also required a solution that is server-, for example either cBioPortal backend- order standalone-tool-based. As one of our main goals was to contribute our extension to the public cBioPortal open-source project, solutions requiring additional computing resources server-sided or manual data updates were not feasible. With the chosen frontend integration of the search tool, all computing is performed in the browser of the end user.

While developing our extension, we focused on a user-centered design 45 with multiple feedback loops to be able to analyze and then support the workflow of the physicians as much as possible. This greatly improved the design and development process as most of the times, translating the requirements of the real-world users to technical features implemented in software is not trivial and should be reviewed and revised when necessary. In summary, the findings of the evaluation including the SUS score of 83.5 showed overall, the tool facilitates the search for matching trials ( Table 1 ).

The extension described in this work is only one of the needed functionalities identified by Buechner et al 15 to use cBioPortal as an MTB platform. The search for clinical trials will be refined using the results of the usability test and gradually other key features will be additionally implemented to supply MTB participants with a comprehensive solution for their routine tasks. The completed prototype for the MTB platform will then be extensively evaluated in a future study.

Limitations

The final usability evaluation was intended to be the last, more detailed round of feedback in the development process, rather than a summative evaluation of a final product. However, the five participants match the often cited threshold to identify the most serious usability issues 46 and provided valuable feedback from real-world users regularly participating in MTBs. The evaluation provides an indication of the usability of the application but needs to be confirmed after the tool has been finalized during the aforementioned extensive evaluation of the completed MTB platform. In particular, the large deviation of SUS scores indicates usability issues that may only apply to a subgroup of users and should be further investigated.

Conclusion

Using a user-centered design process, we developed an integration of the ClinicalTrials.gov registry with cBioPortal. The architecture of the integration is as lightweight as possible, only being coupled with the cBioPortal frontend and requiring no additional server-sided resources. The final evaluation showed a good overall usability and that the tool can assist physicians to find appropriate studies for individual patients in the preparation of MTBs with less manual effort.

Clinical Relevance Statement

This work proposes a tool for searching clinical trials based on molecular alterations in tumors of cancer patients. This facilitates the process of preparing patient data and discussing therapy recommendations in the clinical setting during Molecular Tumor Boards.

Multiple Choice Questions

  • 1. How was the search tool integrated?

    1. In the backend of cBioPortal

    2. In the frontend of cBioPortal

    3. As a standalone service

    4. As a combination of frontend and backend integration

    Correct Answer: The correct answer is option b. The integration of the tool in cBioPortal is frontend-based. As cBioPortal does not provide a plugin concept, implementing the tool in the frontend achieves a loose coupling to the codebase and better compatibility with future updates.

  • 2. What was the main requirement of application?

    1. Transforming trials to a genomics-optimized data format

    2. Facilitate the curation of trial data by experts

    3. Searching trials without the need for manual curation in advance

    4. Enabling patients to search trials in which they could participate

    Correct Answer: The correct answer is option c. The main requirement was that the search for clinical trials should not require any manual curation or preprocessing of trial data and should be executed on the fly. The application should be used by experts in or during the preparation of an MTB.

Acknowledgments

The present work was performed in (partial) fulfillment of the requirements for obtaining the degree “Dr. rer. biol. hum.” from the Friedrich-Alexander-Universität Erlangen-Nürnberg (PU).

Funding Statement

Funding This research has been conducted within the MIRACUM project. MIRACUM is funded by the German Federal Ministry of Education and Research (BMBF), grant IDs 01ZZ1801A (P.U., L.M., and J.C.) and 01ZZ1801B (M.B.). This work was supported by the BMBF funded HiGHmed project, grant ID 01ZZ1802Z (H.B., N.R.) and by the German Research Foundation (DFG), grant IDs CRC 850 and CRC 1160 (M.B.).

Conflict of Interest None declared.

Protection of Human and Animal Subjects

The project was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects. Ethical approval was not required.

*

These authors contributed equally to this work.

References

  • 1.Service R F.Gene sequencing. The race for the $1000 genome Science 2006311(5767):1544–1546. [DOI] [PubMed] [Google Scholar]
  • 2.Shendure J, Ji H. Next-generation DNA sequencing. Nat Biotechnol. 2008;26(10):1135–1145. doi: 10.1038/nbt1486. [DOI] [PubMed] [Google Scholar]
  • 3.Garraway L A, Verweij J, Ballman K V. Precision oncology: an overview. J Clin Oncol. 2013;31(15):1803–1805. doi: 10.1200/JCO.2013.49.4799. [DOI] [PubMed] [Google Scholar]
  • 4.Alves R CP, Alves D, Guz B. Advanced hepatocellular carcinoma. Review of targeted molecular drugs. Ann Hepatol. 2011;10(01):21–27. [PubMed] [Google Scholar]
  • 5.Brehmer D, Greff Z, Godl K. Cellular targets of gefitinib. Cancer Res. 2005;65(02):379–382. [PubMed] [Google Scholar]
  • 6.Arora A, Scholar E M. Role of tyrosine kinase inhibitors in cancer therapy. J Pharmacol Exp Ther. 2005;315(03):971–979. doi: 10.1124/jpet.105.084145. [DOI] [PubMed] [Google Scholar]
  • 7.Cunanan K M, Gonen M, Shen R. Basket trials in oncology: a trade-off between complexity and efficiency. J Clin Oncol. 2017;35(03):271–273. doi: 10.1200/JCO.2016.69.9751. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Redig A J, Jänne P A. Basket trials and the evolution of clinical trial design in an era of genomic medicine. J Clin Oncol. 2015;33(09):975–977. doi: 10.1200/JCO.2014.59.8433. [DOI] [PubMed] [Google Scholar]
  • 9.Arnold D, Bokemeyer C.Studien und personalisierte Medizin in der Onkologie? Oncol Res Treat 201033(Suppl. 7):25–29. [DOI] [PubMed] [Google Scholar]
  • 10.Holch J W, Westphalen C B, Hiddemann W, Heinemann V, Jung A, Metzeler K H. Präzisionsonkologie und molekulare Tumorboards – Konzepte, Chancen und Herausforderungen. Dtsch Med Wochenschr. 2017;142(22):1676–1684. doi: 10.1055/s-0042-120717. [DOI] [PubMed] [Google Scholar]
  • 11.Lindeman N I, Cagle P T, Aisner D L. Updated molecular testing guideline for the selection of lung cancer patients for treatment with targeted tyrosine kinase inhibitors: guideline from the College of American Pathologists, the International Association for the Study of Lung Cancer, and the Association for Molecular Pathology. Arch Pathol Lab Med. 2018;142(03):321–346. doi: 10.5858/arpa.2017-0388-CP. [DOI] [PubMed] [Google Scholar]
  • 12.Penberthy L T, Dahman B A, Petkov V I, DeShazo J P. Effort required in eligibility screening for clinical trials. J Oncol Pract. 2012;8(06):365–370. doi: 10.1200/JOP.2012.000646. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Cerami E, Gao J, Dogrusoz U. The cBio cancer genomics portal: an open platform for exploring multidimensional cancer genomics data. Cancer Discov. 2012;2(05):401–404. doi: 10.1158/2159-8290.CD-12-0095. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Gao J, Aksoy B A, Dogrusoz U. Integrative analysis of complex cancer genomics and clinical profiles using the cBioPortal. Sci Signal. 2013;6(269):pl1–pl1. doi: 10.1126/scisignal.2004088. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Buechner P, Hinderer M, Unberath P. Requirements analysis and specification for a molecular tumor board platform based on cBioPortal. Diagnostics (Basel) 2020;10(02):93. doi: 10.3390/diagnostics10020093. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Lindsay J, Fitz CDV, Zwiesler Z. MatchMiner: An open source computational platform for real-time matching of cancer patients to precision medicine clinical trials using genomic and clinical criteria. bioRxiv. 2017:199489. [Google Scholar]
  • 17.Micheel C M, Lovly C M, Levy M A. My cancer genome. Cancer Genet. 2014;207(06):289. [Google Scholar]
  • 18.Sahoo S S, Tao S, Parchman A. Trial prospector: matching patients with cancer research studies using an automated and scalable approach. Cancer Inform. 2014;13:157–166. doi: 10.4137/CIN.S19454. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.MSKCC . Patient P-0000465. Accessed May 7, 2021 at:https://www.cbioportal.org/patient/summary?studyId=msk_impact_2017&caseId=P-0000465
  • 20.Likert R.A technique for the measurement of attitudesArch Psychol1932
  • 21.Brooke J. SUS-A quick and dirty usability scale. Usability Evaluation in Industry. 1996;189(194):4–7. [Google Scholar]
  • 22.Gao M, Kortum P, Oswald F L. Multi-language toolkit for the system usability scale. Int J Hum Comput Interact. 2020;36(20):1883–1901. [Google Scholar]
  • 23.U.S. National Library of Medicine . ClinicalTrials.gov API. Accessed February 11, 2022 at:https://clinicaltrials.gov/api/gui
  • 24.Unberath P, Knell C, Prokosch H-U, Christoph J. Developing new analysis functions for a translational research platform: extending the cBioPortal for cancer genomics. Stud Health Technol Inform. 2019;258:46–50. [PubMed] [Google Scholar]
  • 25.Bangor A, Kortum P, Miller J. Determining what individual SUS scores mean: adding an adjective rating scale. J Usability Stud. 2009;4(03):114–123. [Google Scholar]
  • 26.Comis R L, Miller J D, Aldigé C R, Krebs L, Stoval E. Public attitudes toward participation in cancer clinical trials. J Clin Oncol. 2003;21(05):830–835. doi: 10.1200/JCO.2003.02.105. [DOI] [PubMed] [Google Scholar]
  • 27.Unger J M, Cook E, Tai E, Bleyer A. The role of clinical trial participation in cancer research: barriers, evidence, and strategies. Am Soc Clin Oncol Educ Book. 2016;35:185–198. doi: 10.14694/EDBK_156686. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Lara P N, Jr., Higdon R, Lim N. Prospective evaluation of cancer clinical trial accrual patterns: identifying potential barriers to enrollment. J Clin Oncol. 2001;19(06):1728–1733. doi: 10.1200/JCO.2001.19.6.1728. [DOI] [PubMed] [Google Scholar]
  • 29.Kaplan C P, Nápoles A M, Dohan D. Clinical trial discussion, referral, and recruitment: physician, patient, and system factors. Cancer Causes Control. 2013;24(05):979–988. doi: 10.1007/s10552-013-0173-5. [DOI] [PubMed] [Google Scholar]
  • 30.Parker B A, Schwaederlé M, Scur M D. Breast cancer experience of the molecular tumor board at the University of California, San Diego Moores Cancer Center. J Oncol Pract. 2015;11(06):442–449. doi: 10.1200/JOP.2015.004127. [DOI] [PubMed] [Google Scholar]
  • 31.Bryce A H, Egan J B, Borad M J. Experience with precision genomics and tumor board, indicates frequent target identification, but barriers to delivery. Oncotarget. 2017;8(16):27145–27154. doi: 10.18632/oncotarget.16057. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Halfmann M, Stenzhorn H, Gerjets P, Kohlbacher O, Oestermeier U. User-Driven Development of a Novel Molecular Tumor Board Support Tool. Accessed February 11, 2022 at:https://events.tib.eu/fileadmin/data/dils2018/paper/paper_6.pdf
  • 33.Fegeler C, Zsebedits D, Bochum S, Finkeisen D, Martens U M. Implementierung eines IT-gestützten molekularen Tumorboards in der Regelversorgung. Forum. 2018;33:322–328. [Google Scholar]
  • 34.Melzer G, Maiwald T, Prokosch H-U, Ganslandt T. Leveraging real-world data for the selection of relevant eligibility criteria for the implementation of electronic recruitment support in clinical trials. Appl Clin Inform. 2021;12(01):17–26. doi: 10.1055/s-0040-1721010. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35.Sun Y, Butler A, Diallo I. A framework for systematic assessment of clinical trial population representativeness using electronic health records data. Appl Clin Inform. 2021;12(04):816–825. doi: 10.1055/s-0041-1733846. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36.O'Leary T, Weiss J, Toll B, Brandt C, Bernstein S L. Automated generation of CONSORT diagrams using relational database software. Appl Clin Inform. 2019;10(01):60–65. doi: 10.1055/s-0038-1677043. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.Dharod A, Bellinger C, Foley K, Case L D, Miller D. The reach and feasibility of an interactive lung cancer screening decision aid delivered by patient portal. Appl Clin Inform. 2019;10(01):19–27. doi: 10.1055/s-0038-1676807. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38.Naceanceno K S, House S L, Asaro P V. Shared-task worklists improve clinical trial recruitment workflow in an academic emergency department. Appl Clin Inform. 2021;12(02):293–300. doi: 10.1055/s-0041-1727153. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 39.Xu J, Lee H-J, Zeng J. Extracting genetic alteration information for personalized cancer therapy from ClinicalTrials.gov. J Am Med Inform Assoc. 2016;23(04):750–757. doi: 10.1093/jamia/ocw009. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40.Zeng J, Shufean M A, Khotskaya Y. OCTANE: oncology clinical trial annotation engine. JCO Clin Cancer Inform. 2019;3:1–11. doi: 10.1200/CCI.18.00145. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 41.Saleh R R, Chan O, Kuld S.A novel electronic platform to improve clinical trial workflow and screeningPresented at AACR Special Conference on Advancing Precision Medicine Drug Development: Incorporation of Real-World Data and Other Novel Strategies; January 9–12: San Diego, CA;2020 [Google Scholar]
  • 42.Lipscomb C E. Medical subject headings (MeSH) Bull Med Libr Assoc. 2000;88(03):265–266. [PMC free article] [PubMed] [Google Scholar]
  • 43.Gene Ontology Consortium . Consortium G O. Creating the gene ontology resource: design and implementation. Genome Res. 2001;11(08):1425–1433. doi: 10.1101/gr.180801. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 44.Wain H M, Bruford E A, Lovering R C, Lush M J, Wright M W, Povey S. Guidelines for human gene nomenclature. Genomics. 2002;79(04):464–470. doi: 10.1006/geno.2002.6748. [DOI] [PubMed] [Google Scholar]
  • 45.Abras C, Maloney-Krichmar D, Preece J. Thousand Oaks, CA: Sage Publications.; 2004. User-centered design. In: Bainbridge W, ed. Encyclopedia of Human-Computer Interaction; pp. 445–456. [Google Scholar]
  • 46.Virzi R A. Refining the test phase of usability evaluation: how many subjects is enough? Hum Factors. 1992;34(04):457–468. [Google Scholar]

Articles from Applied Clinical Informatics are provided here courtesy of Thieme Medical Publishers

RESOURCES