Abstract
Objective
The development of clinical research informatics tools and workflow processes associated with re-engaging biobank participants has become necessary as genomic repositories increasingly consider the return of actionable research results.
Materials and Methods
Here we describe the development and utility of an informatics application for participant recruitment and enrollment management for the Veterans Affairs Million Veteran Program Return Of Actionable Results Study, a randomized controlled pilot trial returning individual genetic results associated with familial hypercholesterolemia.
Results
The application is developed in Python-Flask and was placed into production in November 2021. The application includes modules for chart review, medication reconciliation, participant contact and biospecimen logging, survey recording, randomization, and documentation of genetic counseling and result disclosure. Three primary users, a genetic counselor and two research coordinators, and 326 Veteran participants have been integrated into the system as of February 23, 2023. The application has successfully handled 3367 task requests involving greater than 95 000 structured data points. Specifically, application users have recorded 326 chart reviews, 867 recruitment telephone calls, 158 telephone-based surveys, and 61 return of results genetic counseling sessions, among other available study tasks.
Conclusion
The development of usable, customizable, and secure informatics tools will become increasingly important as large genomic repositories begin to return research results at scale. Our work provides a proof-of-concept for developing and using such tools to aid in managing the return of results process within a national biobank.
Keywords: informatics, genetic testing, genetic databanks, randomized controlled trial
Introduction
Clinical research informatics is a burgeoning domain focused on using information technologies to optimize the design and conduct of translational research.1–3 Integrated informatics tools are vital to improving efficiency over the entire research continuum from eligibility screening through study-related and electronic health records (EHR) outcomes data capture and analytics.4–8 In parallel, the development of workflow processes associated with re-engaging biobank participants has become necessary as genomic repositories increasingly consider the return of actionable genetic information.9–11 The development of such processes might involve myriad factors including genetic variant selection, re-contact and re-consent, biospecimen collection, disclosure processes, the integration of confirmed results into clinical care, and data curation, management, and security, among others.12–17
The US Department of Veterans Affairs (VA) Million Veteran Program (MVP)18 has begun to address some of these ethical, procedural, and technological challenges to explore returning individual research results to its participants. One such initiative is the MVP Return Of Actionable Results (MVP-ROAR) Study, a nationwide randomized controlled pilot trial returning genetic results associated with familial hypercholesterolemia (FH).19 To support this unique pilot study, we developed a flexible informatics tool to aid in the re-contact, re-consent, and enrollment management for one of the first initiatives to engage MVP participants in the process of returning actionable genetic results.
Objective
Here we describe the development and utility of an informatics application for participant recruitment, enrollment management, and data collection for MVP-ROAR. The application serves as a proof-of-concept for streamlining return of results processes and providing real-time status updates for individual enrollees and for the study as a whole. The application utilizes open-source software that can be adapted and scaled to a variety of return of results and clinical trial use cases. Base code for the application is available for review and re-use for other VA and non-VA projects (https://github.com/Genomes2Veterans/mvp-roar-app).
Materials and methods
MVP is one of the largest genetic and health-related research initiatives in the world, having enrolled more than one million participants to date.20 Participants provide a blood specimen, complete health and lifestyle surveys, and give permission for access to their health records. This includes records from the Veterans Health Administration, an integrated health care system comprising nearly 1300 facilities that serves approximately nine million Veterans across the US annually.21 Additionally, as part of enrollment, MVP participants agree to be contacted for future voluntary research activities. They are also informed that their genetic information will not be returned to them nor made available in their health records. Genetic and health data from MVP enrollees have almost exclusively been used to date to conduct large-scale genomic discovery association studies.
To better understand the potential for providing individual-level data back to participants, MVP has launched pilot work designed to re-contact MVP participants for recruitment into return of results research. MVP-ROAR, one of these pilot projects, is exploring the feasibility of returning results to MVP participants and documenting the associated process- and health-related outcomes. MVP-ROAR focuses on the return of genetic information associated with FH, a monogenic disease affecting approximately 1 in 250 adults.22 FH is characterized by severe elevations in low-density lipoprotein cholesterol (LDL-C) and markedly increased risk of premature coronary artery disease.23 The project involves genotype-first identification of MVP participants suspected of harboring FH-related variants. Prior to participant outreach, the MVP-ROAR team works closely with MVP Core research staff to curate eligible FH-related variants, as determined actionable by the Clinical Genome Resource (ClinGen) FH Variant Curation Expert Panel.24 MVP Core research staff use this information to identify eligible MVP participants and conduct initial re-contact efforts to inform them about MVP-ROAR. Participants who do not opt-out of being contacted are recruited by MVP-ROAR, with enrollment involving a separate consent process, genetic result confirmation via next-generation sequencing, and disclosure of the genetic confirmation test and genetic counseling either immediately or after 6-months to a projected 200 participants and their health care providers at VA facilities nationwide. The primary outcome is change in LDL-C from study baseline to end of study (6-months post-enrollment), with the expectation that individuals who receive confirmation of molecular diagnosis of FH (immediate arm) will receive FH-tailored care and experience greater reductions in LDL-C compared to those who receive a genetic confirmation test after 6-months (delayed arm). Additional outcomes and processes will explore proportions of participants meeting LDL-C targets (eg, <100 mg/dL for primary prevention), medication starts, intensification, and adherence, lifestyle modifications (eg, smoking, physical activity), and number of first-degree relatives who undergo cascade screening (provided at no additional cost to participant family members by the genetic testing vendor). Both MVP and MVP-ROAR have been approved by the VA Central Institutional Review Board (IRB) and all Veteran participants provide consent to participate in each research effort. The study integrates MVP, MVP-ROAR, and historical and current clinical data from the nationally deployed VA EHR25 system to initiate and track the entire return of results process and aid in the recording of study-related outcomes.
We leveraged the robust VA Informatics and Computing Infrastructure (VINCI)26 to develop a custom end-to-end informatics solution to support MVP-ROAR. Conception and development of the application began in May 2021 with the first complete version placed into production beginning November 2021. The multi-user application integrates nationwide clinical and administrative data, electronic data capture, and dashboard analytics via a combination of commercial and open-source software. The application is housed within a secure virtual server, is password protected, and accessible only to IRB-approved and system authenticated study staff from within the VINCI environment. Study data from non-opted out participants are queried from VA Corporate Data Warehouse (CDW)27,28 views or entered by study staff and stored within dedicated study tables easing data access for the study team and providing systematic structure and secure storage of trial data for eventual data analysis and sharing. An entity relationship diagram of the entire database structure required for the application is presented in the Supplementary Material. We rely on a task-based workflow structure (Figure 1) to transition participants through the application as return of results and other trial tasks are completed by study staff and participants in real-time.
Figure 1.
Generalized MVP-ROAR application framework and computing environment infrastructure. MVP identifies participants with actionable genetic information and provisions contact and other data to MVP-ROAR Study team after a brief opt-out period. Once data are provisioned, MVP-ROAR staff initiates pilot trial recruitment, enrollment, and return of results process with informatics support via a secure and user-centric web-based application. Abbreviations: CDW, Corporate Data Warehouse; MVP, Million Veteran Program; ROAR; Return of Actionable Results; VINCI, VA Informatics and Computing Infrastructure.
The application is developed in Python (v3.8.14)29 and uses the Flask micro web framework (v2.0.0)30,31 along with libraries supporting data extraction, manipulation, and visualization,32–36 to provide study staff with detailed views, task lists, and data entry and editing capabilities via a standard web browser. Relevant software and supporting packages required for the application are presented in Table 1. Development and maintenance of the application is managed by two members of the study team entirely within a VINCI-supported development environment. Initial development of the application was completed by one part-time developer (∼0.25 FTE) over approximately six months. General management of the fully developed application, including application and database maintenance, requires approximately 2 hours per week of staff time. Patient-facing study staff with access to the application include a board-certified genetic counselor and two research coordinators.
Table 1.
MVP-ROAR application development software and package infrastructure.
| Application system | Software and supporting packages |
|---|---|
| Database system |
|
| Server |
|
| Backend software and development infrastructure |
|
| Frontend software and development infrastructure |
|
Standard libraries and package dependencies not listed.
Layouts templated using Jinja2 (v3.0.1).
Results
Main application components include a study dashboard, searchable cohort and task lists, and individual participant views and data entry modules (Figure 2). The dashboard presents users with aggregate recruitment and enrollment information as well as participant demographic, geographic, and outcomes data. Dashboard elements are fully customizable and are prioritized to offer meaningful and real-time snapshots of study progress for daily operations as well as for scheduled operational and scientific team meetings. The application includes two distinct cohort views, one including all participants integrated into the system and a second structured by pending task whereby the research team is able to view participants according to their current status in the overall trial workflow. Individual task modules as well as demographic and contact information, relevant clinical, medication, and health care data, and an action and contact log are accessible for each individual via a participant detail page. Workflow modules are presented to application users in sequential order and display task status (pending or complete), task due date, and task completion date for efficient reference. Each task module listed in the workflow includes a button to route study staff to a corresponding web-based module for data entry or task management. Task module buttons are only made available to study staff if the prior workflow task has been completed and validated, ensuring proper transition of participants through the study workflow.
Figure 2.
MVP-ROAR application user views. (A) The study dashboard including aggregate recruitment and enrollment metrics. (B) A participant detail and workflow task list available to study staff. (C) Participant demographic and study status update module.
Application components, task modules, and data entry and editing features built specifically for the MVP-ROAR application are presented in Table 2. Information from each completed task, data entry, or data edit associated with an individual participant is logged in the study database as an “action” and presented to application users via an action log included on each participant’s detail page. A staff identification number and timestamp are recorded for each action in the study database in addition to task-specific information for auditing purposes. The action log provides application users with historical information, including attempted contacts, participant notes, and task completion information by calendar date for each MVP-ROAR participant beginning with MVP data provisioning through end-of-study incentive payment ($50 for completion of end-of-study survey and biospecimen collection).
Table 2.
MVP-ROAR application study views, data entry, and workflow modules.
| Application component | Description |
|---|---|
| Dashboard | Presents aggregated data associated with recruitment, enrollment, and randomization status, basic demographic information (participant sex, race, and geographic representativeness), and DNA specimen type (blood or saliva) and result confirmation outcomes. |
| Cohort view | Presents a table of all potential and enrolled study participants by study ID, name, date of birth, and study status, among other quick reference data. |
| Task view | Presents a filterable table of participants by study workflow task. Only active (alive, currently being recruited, or consented and enrolled) participants are included in this view. |
| Chart review module | A data entry and validation form for manual EHR chart review by study staff. Chart review outcomes aid the study team in determining eligibility status for each potential MVP-ROAR Study participant. |
| Recruitment letter tracking | A data entry tool for recording recruitment letter status and date of letter distribution. |
| Contact and consent tracking | A data entry tool to log each participant telephone or email communication with the study team throughout participants’ entire study enrollment. |
| Pedigree collection tracking | A data entry tool for recording whether a participant has provided the study genetic counselor with relevant family history information. Pedigree information is collected to appropriately frame/tailor each participant’s genetic counseling session and note. |
| Biospecimen collection tracking | A data entry tool for recording whether participants’ study specimens (either blood or saliva at baseline and end-of-study) have been received by the study team. |
| LDL-C tracking | A data entry tool for recording participant LDL-C measurements (maximum from EHR, baseline study draw, end-of-study draw). LDL-C measurements are manually entered into the system by study staff. |
| Genetic report, intervention delivery, and intervention documentation tracking | A data entry tool for recording receipt of genetic confirmation report from reference laboratory, recording of results delivery and mechanism (phone, video, etc.), and that proper documentation has been delivered to participant, preferred participant health care provider, and entered in EHR. |
| Incentive tracking | A data entry tool for recording end-of-study participant incentive distribution. |
| Baseline and end-of-study survey administration | A data entry tool for study staff to record baseline and end-of-study surveys administered by telephone. Surveys include branching and other logic for differential survey instruments administered to immediate versus delayed genetic results recipients. |
| Randomization module | A tool for research staff to randomize study participants upon the receipt of baseline specimens. The tool utilizes both user input and data validation from the study database to ensure randomization may occur at the time the tool is used. Study team is immediately presented with the outcome and status is committed to the study database in real-time. |
| Participant information and status update module | A data entry tool to enter and edit participant contact information, preferred VA facility, preferred health care provider information, and unable to contact, loss to follow up, and study withdrawal status. |
| Medication reconciliation module | A data entry tool to record and edit current participant medications either from the EHR or directly from participants. |
| Action and contact log | Presents a date sorted record of all contact and study-related actions associated with each MVP-ROAR Study participant. Intended to provide research staff with a quick reference of the entire recruitment and enrollment history. |
Abbreviations: EHR, electronic health record; LDL-C, low-density lipoprotein cholesterol; MVP, Million Veteran Program; ROAR, Return of Actionable Results; VA, Veterans Affairs.
A total of 326 Veteran participants have been integrated into the system as of February 23, 2023. Overall, 3 primary users, a genetic counselor and two research coordinators, have used the application to successfully handle 3367 individual task requests involving greater than 95 000 structured data points associated with participant tracking, operational efficiency (eg, fields for staff to record VA facility specific information or to include time spent completing tasks for eventual econometric assessment), and outcomes data collection (eg, study-related LDL-C measurements or telephone survey response data). Figure 3A presents a distribution of study-specific tasks completed by users from within the MVP-ROAR application between November 2021 and February 2023.
Figure 3.
(A) Total return of results workflow and trial tasks completed and recorded by study staff within the MVP-ROAR application between November 2021 and February 2023. (B) Date of application implementation (November 2021) and cumulative enrollment of trial participants between July 2020 and February 2023. (C) The average number of tasks per active participant per month and the maximum number of tasks per participant per month completed and recorded between November 2021 and February 2023. Abbreviations: EHR, electronic health record; LDL-C, low-density lipoprotein cholesterol.
Operationally, transition to the application from a single-file tracking mechanism coincided with the refinement of variant selection methods and full project re-opening during the COVID-19 pandemic. Overall, the application aided the study team in re-initiating participant recruitment and has helped improve trial enrollment outcomes post-implementation (5.3 vs 3.8 average trial enrollees per month, not including periods of shutdown) (Figure 3B). From an efficiency perspective, users demonstrated acceptable usage of the application completing, on average, just over two tasks per active participant per month as well as recording greater than ten completed tasks associated with single participants across multiple months (Figure 3C). Data from each task are stored in a structured format and can be queried to inform future operational assessments associated with return of results in MVP.
Discussion
The development of usable, customizable, and secure informatics tools is becoming increasingly necessary as large genomic repositories have begun to consider the return of research results at scale.37–39 While other organizations have developed informatics systems to automate pharmacogenomic clinical decision support,40 for direct-to-patient genetic result disclosure,41 or for complex data management and analytics systems,42 few out-of-the-box options currently exist for workflow development and data capture involving an integrated return of results process. While some off-the-shelf products provide user-friendly and efficient informatics support for translational research generally,43 functionality can be limited in some computing and regulatory environments such as VA.44 Moreover, the development of a one-size-fits-all informatics solution may not satisfy the needs of individual institutions participating in genetic return of results, given large heterogeneity in workflow preferences, differences in patient populations and disease focus, and variability in EHR and data systems.45
Specifically, our return of results workflow and clinical trial processes require both systematic structure and individualized detail and flexibility given the translational nature of MVP-ROAR and our aim to accommodate each participant’s unique personal and clinical situations. Thus, our custom-built MVP-ROAR application seeks to aid our staff in this endeavor as we navigate a first-of-its-kind initiative within MVP and the VA on a national scale. We anticipate that final process- and health-related outcomes associated with MVP-ROAR, including those learned from the development and use of our informatics system, will be informative to MVP and biobanks elsewhere for future return of results initiatives.
Primary strengths of the application, like other research and operational applications within VA,46,47 include its security, scope, and integration with EHR data housed within the CDW. One major advantage of CDW data integration involves its standardized structure across VA facilities nationally, including the merging of our application data with EHR data conformant to the Observational Medical Outcomes Partnership Common Data Model.48,49 Additionally, the Python-Flask framework is highly transferable, modifiable, and, depending on project requirements, can be developed, scaled, and managed by teams large or small, particularly with the availability of a well-developed and supported computing infrastructure such as VINCI.50,51 The application offers focused and intuitive data entry interfaces and can handle multiple users as well as multiple database commit and query requests simultaneously, which provides major advantages over single file-based participant tracking mechanisms.52 Conceptually, the development team has greatly benefitted from input on workflow, process, and essential data elements from our interdisciplinary team of physician-scientists, research administrators, genetic counselors, data scientists, and frontline research staff.
A limitation of the application is that it requires a computing infrastructure that can securely and efficiently host web applications and some back- and front-end web development capability from members of the study team. Secondly, the application relies heavily on data entry by study staff and is not push-enabled with the VA EHR, requiring entry of genetic counseling notes and other clinical information directly into the patient medical record in order to bridge the research to clinical care continuum. Additionally, database creation, maintenance, and handling of data entry errors necessitates manual editing directly within the database by the development team, which requires additional study staff competencies in database management.
Conclusion
Our work provides a proof-of-concept for developing and using a custom informatics tool to aid in managing a return of results workflow and clinical trial process via actionable health data from a national biobank. We demonstrate the flexibility and utility of our application and provide our conceptual approach and base application code to assist other teams seeking to implement an informatics solution for returning genetic results to research participants or for conducting clinical trials in a comprehensive and efficient manner.
Supplementary Material
Acknowledgments
We thank all the Veterans and MVP staff we have engaged with throughout the conduct of the MVP-ROAR Study. The views expressed here are those of the authors and do not necessarily reflect the position of the US Department of Veterans Affairs or the US Government.
Contributor Information
Charles A Brunette, Veterans Affairs Boston Healthcare System, Boston, MA, United States; Department of Medicine, Harvard Medical School, Boston, MA, United States.
Thomas Yi, Veterans Affairs Boston Healthcare System, Boston, MA, United States.
Morgan E Danowski, Veterans Affairs Boston Healthcare System, Boston, MA, United States.
Mark Cardellino, Veterans Affairs Boston Healthcare System, Boston, MA, United States.
Alicia Harrison, Genetic Counseling Program, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States.
Themistocles L Assimes, VA Palo Alto Health Care System, Palo Alto, CA, United States; Division of Cardiovascular Medicine, Department of Medicine, Stanford University School of Medicine, Palo Alto, CA, United States; Stanford Cardiovascular Institute, Stanford University, Palo Alto, CA, United States; Department of Epidemiology and Population Health, Stanford University School of Medicine, Palo Alto, CA, United States.
Joshua W Knowles, Division of Cardiovascular Medicine, Department of Medicine, Stanford University School of Medicine, Palo Alto, CA, United States; Stanford Cardiovascular Institute, Stanford University, Palo Alto, CA, United States; Family Heart Foundation, Pasadena, CA, United States.
Kurt D Christensen, PRecisiOn Medicine Translational Research (PROMoTeR) Center, Department of Population Medicine, Harvard Pilgrim Health Care Institute, Boston, MA, United States; Department of Population Medicine, Harvard Medical School, Boston, MA, United States.
Amy C Sturm, 23andMe, Sunnyvale, CA, United States.
Yan V Sun, Atlanta VA Health Care System, Decatur, GA, United States; Department of Epidemiology, Emory University Rollins School of Public Health, Atlanta, GA, United States.
Qin Hui, Atlanta VA Health Care System, Decatur, GA, United States; Department of Epidemiology, Emory University Rollins School of Public Health, Atlanta, GA, United States.
Saiju Pyarajan, Veterans Affairs Boston Healthcare System, Boston, MA, United States; Department of Medicine, Harvard Medical School, Boston, MA, United States.
Yunling Shi, Veterans Affairs Boston Healthcare System, Boston, MA, United States.
Stacey B Whitbourne, Veterans Affairs Boston Healthcare System, Boston, MA, United States; Department of Medicine, Harvard Medical School, Boston, MA, United States; Division of Aging, Department of Medicine, Brigham and Women’s Hospital, Boston, MA, United States.
J Michael Gaziano, Veterans Affairs Boston Healthcare System, Boston, MA, United States; Department of Medicine, Harvard Medical School, Boston, MA, United States; Division of Aging, Department of Medicine, Brigham and Women’s Hospital, Boston, MA, United States.
Sumitra Muralidhar, Office of Research and Development, Veterans Health Administration, Washington, DC, United States.
Jason L Vassy, Veterans Affairs Boston Healthcare System, Boston, MA, United States; Department of Medicine, Harvard Medical School, Boston, MA, United States; Division of General Internal Medicine and Primary Care, Brigham and Women’s Hospital, Boston, MA, United States; Population Precision Health, Ariadne Labs, Boston, MA, United States.
Author contributions
Charles A. Brunette, Morgan E. Danowski, Alicia Harrison, Themistocles L. Assimes, Kurt D. Christensen, Joshua W. Knowles, Amy C. Sturm, Yan V. Sun, and Jason L. Vassy conceptualized and refined the return of results process. Charles A. Brunette, Morgan E. Danowski, and Alicia Harrison conceptualized the application. Charles A. Brunette led application development. Charles A. Brunette and Thomas Yi implemented and managed the application and supporting infrastructure. Morgan E. Danowski and Mark Cardellino tested and used the application to fulfill study requirements. Yan V. Sun, Qin Hui, Saiju Pyarajan, Yunling Shi, Stacey B. Whitbourne, J. Michael Gaziano, and Sumitra Muralidhar provided important MVP-related input and support for project and application development. Jason L. Vassy acquired funding and oversaw the project team. Charles A. Brunette and Thomas Yi drafted the manuscript, and all authors critically reviewed the manuscript.
Supplementary material
Supplementary material is available at JAMIA Open online.
Funding
This research was supported by the Million Veteran Program, Office of Research and Development, US Department of Veterans Affairs.
Conflicts of interest
J.G. reports a relationship with Novartis that includes grant funding. All other authors report no competing interests.
Data availability
Code and documentation for the MVP-ROAR application can be found at https://github.com/Genomes2Veterans/mvp-roar-app. Data underlying application metrics and visualizations presented in this manuscript have not been deposited in a public repository because most of the data include participant-related identifiable information. Deidentified data may be provided from the corresponding author upon reasonable request.
References
- 1. Kahn MG, Weng C.. Clinical research informatics: a conceptual perspective. J Am Med Inform Assoc. 2012;19(e1):e36-e42. 10.1136/amiajnl-2012-000968 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Weng C, Kahn MG.. Clinical research informatics for big data and precision medicine. Yearb Med Inform. 2016;25(1):211-218. 10.15265/IY-2016-019 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Daniel C, Tannier X, Kalra D.. Clinical research informatics. Yearb Med Inform. 2022;31(1):161-164. 10.1055/s-0042-1742530 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. Kondylakis H, Claerhout B, Keyur M, et al. The INTEGRATE project: delivering solutions for efficient multi-centric clinical research and trials. J Biomed Inform. 2016;62:32-47. 10.1016/j.jbi.2016.05.006 [DOI] [PubMed] [Google Scholar]
- 5. Jain NM, Culley A, Knoop T, Micheel C, Osterman T, Levy M.. Conceptual framework to support clinical trial optimization and end-to-End enrollment workflow. JCO Clin Cancer Inform. 2019;3(3):1-10. 10.1200/CCI.19.00033 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Dhayne H, Kilany R, Haque R, Taher Y.. EMR2vec: bridging the gap between patient data and clinical trial. Comput Ind Eng. 2021;156:107236. 10.1016/j.cie.2021.107236 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Rogers JR, Lee J, Zhou Z, Cheung YK, Hripcsak G, Weng C.. Contemporary use of real-world data for clinical trial conduct in the United States: a scoping review. J Am Med Inform Assoc. 2021;28(1):144-154. 10.1093/jamia/ocaa224 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. von Itzstein MS, Hullings M, Mayo H, Beg MS, Williams EL, Gerber DE.. Application of information technology to clinical trial evaluation and enrollment: a review. JAMA Oncol. 2021;7(10):1559-1566. 10.1001/jamaoncol.2021.1165 [DOI] [PubMed] [Google Scholar]
- 9. Jarvik GP, Amendola LM, Berg JS, et al. Return of genomic results to research participants: the floor, the ceiling, and the choices in between. Am J Hum Genet. 2014;94(6):818-826. 10.1016/j.ajhg.2014.04.009 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. Bombard Y, Brothers KB, Fitzgerald-Butt S, et al. The responsibility to recontact research participants after reinterpretation of genetic and genomic research results. Am J Hum Genet. 2019;104(4):578-595. 10.1016/j.ajhg.2019.02.025 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. Lewis ACF, Knoppers BM, Green RC.. An international policy on returning genomic research results. Genome Med. 2021;13(1):115. 10.1186/s13073-021-00928-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. Wong CA, Hernandez AF, Califf RM.. Return of research results to study participants: uncharted and untested. JAMA. 2018;320(5):435-436. 10.1001/jama.2018.7898 [DOI] [PubMed] [Google Scholar]
- 13. Arshad S, Arshad J, Khan MM, Parkinson S.. Analysis of security and privacy challenges for DNA-genomics applications and databases. J Biomed Inform. 2021;119:103815. 10.1016/j.jbi.2021.103815 [DOI] [PubMed] [Google Scholar]
- 14. Blout Zawatsky CL, Shah N, Machini K, et al. Returning actionable genomic results in a research biobank: analytic validity, clinical implementation, and resource utilization. Am J Hum Genet. 2021;108(12):2224-2237. 10.1016/j.ajhg.2021.10.005 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15. Raymond MB, Cooper KE, Parker LS, Bonham VL.. Practices and attitudes toward returning genomic research results to Low-Resource research participants. Public Health Genomics. 2021;24(5-6):241-252. 10.1159/000516782 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Vears DF, Minion JT, Roberts SJ, et al. Return of individual research results from genomic research: a systematic review of stakeholder perspectives. PLoS One. 2021;16(11):e0258646. 10.1371/journal.pone.0258646 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17. Kawame H, Fukushima A, Fuse N, et al. The return of individual genomic results to research participants: design and pilot study of tohoku medical megabank project. J Hum Genet. 2022;67(1):9-17. 10.1038/s10038-021-00952-8 [DOI] [PubMed] [Google Scholar]
- 18. Gaziano JM, Concato J, Brophy M, et al. Million veteran program: a mega-biobank to study genetic influences on health and disease. J Clin Epidemiol. 2016;70:214-223. 10.1016/j.jclinepi.2015.09.016 [DOI] [PubMed] [Google Scholar]
- 19. VA Office of Research and Development. Million Veteran Program Return of Actionable Results. clinicaltrials.gov; 2022. Accessed December 29, 2022. https://clinicaltrials.gov/ct2/show/NCT04178122
- 20. Million Veteran Program (MVP). Published November 7, 2022. Accessed January 3, 2023. https://www.research.va.gov/mvp/
- 21.Veterans Health Administration. Accessed January 3, 2023. https://www.va.gov/health/
- 22. Sun YV, Damrauer SM, Hui Q, et al. Effects of genetic variants associated with familial hypercholesterolemia on low-density lipoprotein-cholesterol levels and cardiovascular outcomes in the Million Veteran Program. Circ Genom Precis Med. 2018;11(12):e002192. 10.1161/CIRCGEN.118.002192 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23. Tokgozoglu L, Kayikcioglu M.. Familial hypercholesterolemia: global burden and approaches. Curr Cardiol Rep. 2021;23(10):151. 10.1007/s11886-021-01565-5 [DOI] [PubMed] [Google Scholar]
- 24. Chora JR, Iacocca MA, Tichý L, et al. The Clinical Genome Resource (ClinGen) Familial Hypercholesterolemia Variant Curation Expert Panel consensus guidelines for LDLR variant classification. Genet Med. 2022;24(2):293-306. 10.1016/j.gim.2021.09.012 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25. Deckro J, Phillips T, Davis A, Hehr AT, Ochylski S.. Big data in the veterans health administration: a nursing informatics perspective. J Nurs Scholarsh. 2021;53(3):288-295. 10.1111/jnu.12631 [DOI] [PubMed] [Google Scholar]
- 26. Velarde KE, Romesser JM, Johnson MR, et al. An initiative using informatics to facilitate clinical research planning and recruitment in the VA health care system. Contemp Clin Trials Commun. 2018;11:107-112. 10.1016/j.conctc.2018.07.001 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27. Fihn SD, Francis J, Clancy C, et al. Insights from advanced analytics at the veterans health administration. Health Affairs (Millwood). 2014;33(7):1203-1211. [DOI] [PubMed] [Google Scholar]
- 28. Price LE, Shea K, Gephart S.. The veterans affairs’s corporate data warehouse: uses and implications for nursing research and practice. Nurs Adm Q. 2015;39(4):311-318. 10.1097/NAQ.0000000000000118 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29. Python 3.8.14 Documentation. Accessed December 29, 2022. https://docs.python.org/3.8/
- 30. Flask Documentation (2.0.x). Accessed December 29, 2022. https://flask.palletsprojects.com/en/2.0.x/
- 31. Grinberg M. Flask Web Development: Developing Web Applications with Python. O’Reilly Media, Inc.; 2018. [Google Scholar]
- 32. Bootstrap. Accessed December 29, 2022. https://getbootstrap.com/
- 33. pandas—Python Data Analysis Library. Accessed December 29, 2022. https://pandas.pydata.org/
- 34. Plotly. Accessed December 29, 2022. https://plotly.com/python/
- 35. Copeland R. Essential Sqlalchemy. O’Reilly Media, Inc.; 2008. [Google Scholar]
- 36. Harris CR, Millman KJ, van der Walt SJ, et al. Array programming with NumPy. Nature. 2020;585(7825):357-362. 10.1038/s41586-020-2649-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37. NIH’s All of Us Research Program returns first genetic results to participants. All of Us Research Program | NIH. Published December 4, 2020. Accessed January 10, 2023. https://allofus.nih.gov/news-events/announcements/nihs-all-us-research-program-returns-first-genetic-results-participants
- 38. Return of Results | Mass General Brigham. Accessed January 10, 2023. https://www.massgeneralbrigham.org/en/research-and-innovation/participate-in-research/biobank/biobank-and-you/return-of-research-results
- 39. Return of Results. Accessed January 10, 2023. https://medschool.cuanschutz.edu/cobiobank/return-of-results
- 40. Liu M, Vnencak-Jones CL, Roland BP, et al. A tutorial for pharmacogenomics implementation through end-to-end clinical decision support based on ten years of experience from PREDICT. Clin Pharmacol Ther. 2021;109(1):101-115. 10.1002/cpt.2079 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41. Biesecker BB, Lewis KL, Umstead KL, et al. Web platform vs in-Person genetic counselor for return of carrier results from exome sequencing: a randomized clinical trial. JAMA Intern Med. 2018;178(3):338-346. 10.1001/jamainternmed.2017.8049 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42. Das S, Zijdenbos AP, Harlap J, Vins D, Evans AC.. LORIS: a web-based data management system for multi-center studies. Front Neuroinform. 2011;5:37. 10.3389/fninf.2011.00037 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG.. Research electronic data capture (REDCap) - a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(2):377-381. 10.1016/j.jbi.2008.08.010 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44. Paris BL, Hynes DM.. Diffusion, implementation, and use of research electronic data capture (REDCap) in the veterans health administration (VA). JAMIA Open. 2019;2(3):312-316. 10.1093/jamiaopen/ooz017 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45. Wiesner GL, Kulchak Rahm A, Appelbaum P, et al. Returning results in the genomic era: Initial experiences of the eMERGE network. J Pers Med. 2020;10(2):30. 10.3390/jpm10020030 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46. Fillmore NR, Elbers DC, La J, et al. An application to support COVID-19 occupational health and patient tracking at a veterans affairs medical center. J Am Med Inform Assoc. 2020;27(11):1716-1720. 10.1093/jamia/ocaa162 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47. Dhond R, Elbers D, Majahalme N, et al. ProjectFlow: a configurable workflow management application for point of care research. JAMIA Open. 2021;4(3):ooab074. 10.1093/jamiaopen/ooab074 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 48. Lynch KE, Deppen SA, DuVall SL, et al. Incrementally transforming electronic medical records into the observational medical outcomes partnership common data model: a multidimensional quality assurance approach. Appl Clin Inform. 2019;10(5):794-803. 10.1055/s-0039-1697598 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 49. Viernes B, Lynch KE, South B, Coronado G, DuVall SL.. Characterizing VA users with the OMOP common data model. Stud Health Technol Inform. 2019;264:1614-1615. 10.3233/SHTI190561 [DOI] [PubMed] [Google Scholar]
- 50. Cuddihy T, Forde B, Rhodes N, et al. SRA down under: Cache and analysis platform for infectious disease. Stud Health Technol Inform. 2019;266:76-82. 10.3233/SHTI190776 [DOI] [PubMed] [Google Scholar]
- 51. Maharathi B, Wong J, Geraghty JR, et al. Multi-modal data integration platform combining clinical and preclinical models of post subarachnoid hemorrhage epilepsy. Annu Int Conf IEEE Eng Med Biol Soc. 2022;2022:3459-3463. 10.1109/EMBC48229.2022.9871864 [DOI] [PubMed] [Google Scholar]
- 52. Staziaki PV, Kim P, Vadvala HV, Ghoshhajra BB.. Medical registry data collection efficiency: a crossover study comparing Web-Based electronic data capture and a standard spreadsheet. J Med Internet Res. 2016;18(6):e141. 10.2196/jmir.5576 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
Code and documentation for the MVP-ROAR application can be found at https://github.com/Genomes2Veterans/mvp-roar-app. Data underlying application metrics and visualizations presented in this manuscript have not been deposited in a public repository because most of the data include participant-related identifiable information. Deidentified data may be provided from the corresponding author upon reasonable request.