Abstract
Study objective
Cancer and heart disease are leading causes of mortality, and cardio-oncology is emerging as a new field addressing the cardiovascular toxicities related to cancer and cancer therapy. Interdisciplinary research platforms that incorporate digital health to optimize cardiovascular health and wellness in cancer survivors are therefore needed as we advance in the digital era. Our goal was to develop the Connected Health Innovation Research Program (C.H.I.R.P.) to serve as a foundation for future integration and assessments of adoption and clinical efficacy of digital health tools for cardiovascular health and wellness in the general population and in oncology patients.
Design/setting/participants
Partner companies were identified through the American Medical Association innovation platform, as well as LinkedIn and direct contact by our team. Company leaders met with our team to discuss features of their technology or software. Non-disclosure agreements were signed and data were discussed and obtained for descriptive or statistical analysis.
Results
A suite of companies with technologies focused on wellness, biometrics tracking, audio companions, oxygen saturation, weight trends, sleep patterns, heart rate variability, electrocardiogram patterns, blood pressure patterns, real-time metabolism tracking, instructional video modules, or integration of these technologies into electronic health records was collated. We formed an interdisciplinary research team and established an academia-industry collaborative foundation for connecting patients with wellness digital health technologies.
Conclusions
A suite of software and device technologies accessible to the cardiology and oncology population has been established and will facilitate retrospective, prospective, and case research studies assessing adoption and clinical efficacy of digital health tools in cardiology/oncology.
Keywords: Digital health, Connected health, Innovation, Cardio-oncology, Digital transformation, Learning Health System
1. Introduction
Cardiovascular disease (CVD) is the leading cause of death globally [1]. Cancer is also a leading cause of mortality. Annually, approximately 2 million new cancer diagnoses are made, and more than 600,000 cancer deaths occur [2]. Moreover, in cancer survivors, CVD is a leading cause of death second only to cancer recurrence or the development of new cancers [3]. Cancer therapies are evolving rapidly to improve outcomes for cancer survivors, but many of these therapies associate with cardiovascular toxicities [4]. Thus, the new medical subspecialty cardio-oncology has emerged to address the prevention, monitoring, and management of cardiovascular toxicities related to cancer therapies [5,6]. Yet, studies indicate that cancer survivors may not be receiving optimal monitoring for cardiovascular health and wellness [7,8]. Interestingly, lifestyle behaviors such as physical inactivity are involved in 80 % of the risk for CVD [1], and are also linked to the development of cancer [3,9], as well as cardiovascular outcomes in cancer survivors [3,10]. Continued innovation in monitoring and targeting these health behaviors may help curb the impact of both CVD and cancer.
Connecting novel technological research with medical and academic institutions can further collaborative innovation. Clinicians and researchers in cardiovascular medicine are beginning to embrace the use of artificial intelligence (AI) to incorporate more precise and personalized care for patients [11]. MEDx (Medicine + Engineering at Duke University) was created in 2015 to emphasize interdisciplinary collaborations between the School of Medicine and the School of Engineering, to exchange and create research opportunities and ideas [12]. The goal of this program was to promote collaborations among clinical, academic, and engineering researchers to develop solutions together, through effective technological devices, therapeutics, and care delivery systems [12]. Recognizing the need for guidance and best practices for such collaborations, the American College of Cardiology (ACC) released the first-ever framework to guide cardiovascular technology companies on the manufacturing, evaluation and reporting of data for technologies designed to improve cardiovascular health. The “Best Practices for Consumer Cardiovascular Technology Solutions” standards document was released in January 2022 and was created in collaboration with industry leaders to help physicians start to incorporate such products into clinical practice in an evidence-based way [13]. The Food and Drug Administration (FDA) has also released a draft guidance document in relation to digital health technologies [14]. This document, also released in January 2022, provides non-binding recommendations to sponsors, investigators, and other stakeholders on the use of digital health technologies to acquire clinical investigation data remotely. While not directly relating to physician guidance on implementing digital health, this document will help direct the research necessary to do so. In addition, the FDA Digital Health Center of Excellence was established to advance science by providing evidence for digital health technologies to meet the needs of stakeholders. Nevertheless, practical evidence and studies regarding cardiovascular health and wellness particularly in the oncology population are limited, with sparse guidance for physicians in cardio-oncology to make strong recommendations for particular technologies and act on the data these technologies provide [15].
There is a rapidly expanding body of evidence supporting the use of wearable devices, mobile health applications, and other technologies in digital health. Physiological data such as resting heart rate (RHR) and step counts can play a valuable role in lifestyle and wellness monitoring for an array of health concerns, such as obesity, which are relevant to both heart disease and cancer [16]. These innovative tools for physiological monitoring serve as supplemental data sources that will complement electronic health record (EHR), registry, and claims data to provide a more comprehensive view of patients' health in and outside of the healthcare system [17]. The American Heart Association (AHA) statements on the Learning Healthcare System support the incorporation of digital health in data-driven medicine to improve outcomes and efficiency [17,18]. Biometric data such as information from electrocardiogram (ECG), heart rate (HR), blood pressure (BP), blood glucose (BG), and oxygen saturation (SpO2) collected through real-time feedback via smartphone-linked wearable sensors are gathered and uploaded to healthcare clouds, enabling medical professionals and caregivers access anytime and anywhere, mitigating barriers of geographical location [19]. It is paramount to offer individuals options to choose wearables, mobile applications, and equipment for monitoring stress and sleep, that support evidence of effectiveness and suit their personal needs [20]. However, the use of consumer-grade wearable devices for use beyond fitness tracking remains largely untapped in the general population and particularly in cardio-oncology. Many questions remain unanswered, including how to best implement digital interfaces to bridge the gap between healthcare delivery and the rapid development of health technologies [21,22].
We hypothesized that the Connected Health Innovation Research Program (C.H.I.R.P.) could be developed to integrate innovation, clinician and patient education, and technological solutions to rising cardiovascular health and wellness needs of cancer survivors. C.H.I.R.P. was therefore developed as a bridge between innovators' health and wellness products and individuals in cardiology and oncology, especially cancer survivors. The technologies can potentially improve survivors' health and wellness and clinician's preparedness to incorporate digital health in their care.
2. Materials and methods
2.1. Innovation company screening and recruitment
Program partner companies were identified by our lead attending physician (SAB) primarily by forming connections through the American Medical Association (AMA) innovation platform or on LinkedIn (Fig. 1). Companies were first identified for the fit of their product and potential for improving outcomes in cardio-oncology patients. Collaborating companies were recruited to focus on wearables, mobile health, biometrics, patient education, virtual care platforms, ECG/BP monitors, and other applications of AI in digital health (Fig. 2). Specific applications within these areas differed greatly, with wide applicability (Fig. 3A). Company officers met with our lead attending physician virtually to discuss features of their technology, software, or device (Fig. 1).
2.2. Innovation research team
With a cadre of initial companies established, focus shifted towards coordinating an innovation research team. An interdisciplinary research team was established in partnership with a second attending physician and a medical student eager to connect digital advancement to the improvement of cardio-oncology patient care. The team was created through connections on LinkedIn, organic conversations, email outreach to medical students, and other forms of professional outreach. The interdisciplinary and multilevel team ultimately included physicians and scientists, medical and graduate students, innovators, and entrepreneurs and cardio-oncology patients (Fig. 3B).
2.3. Innovation relationships and confidentiality
Patient-centered research documentation was developed and signed with each company, typically a non-disclosure agreement (NDA), along with a Data Use Agreement (DUA) and/or Business Associate Agreement (BAA) if indicated. NDA agreements were considered standard after initial team meetings with companies to discuss their technologies and potential opportunities for clinical research together. DUA and BAA agreements were commonly initiated by the company or data providers as needed, as a measure of security if outlined in their company data use policies. Our workflow for this entire process is summarized in Fig. 4, which describes steps taken to build C.H.I.R.P.. These steps ranged from establishing our relationship with a company to finalization of plans for preliminary data collection, sharing, and analysis.
2.4. Innovation research foundation
After preliminary meetings of the newly formed research team with each company, an Institutional Review Board (IRB) application was completed and approved, to first start with planning for retrospective studies of general users of the company technologies. Our group then prepared retrospective study proposals that consolidated each company's pitch and our study design. This proposal was generally comprised of the following elements:
-
1.
Research questions: We developed guided questions to gain insight into the utility of the technology/software for each company, such as how easily it was adopted by users, how clinicians and patients could use the data to improve physiological monitoring and supervision, and how adhering to the use of the technology could improve short-term (or long-term) clinical outcomes.
-
2.
Study objectives: We identified two major objectives for our study with each company – assessment of the adoption of and satisfaction with the technology/software as well as user interaction with the technology/software, and assessment of short-term clinical outcomes. Company leaders also expressed their objectives for working with us to build this new research program. Many company founders and representatives expressed specific aims in conducting such research including following guidelines that could support future FDA approval or focusing on a specific subset of patients.
-
3.
Study plan: A description of how we would test our objectives, which included descriptions of the metrics to be used to gauge patient adoption and satisfaction, as well as short-term clinical outcomes (e.g., BP, HR, ECG) that the technology or software measures was made collaboratively, based on the kind of data available from each company.
A subsequent meeting scheduled with each company was used to share the study proposal. This ensured collaborative awareness and input on our proposed study design for data use. These initial proposed studies were designed to retrospectively review deidentified data from general users of C.H.I.R.P. partner technologies and software to assess: adoption and patterns of use as well as user satisfaction, and the effects of partner technologies and software on short-term clinical outcomes. Our intention was to use findings from these retrospective studies on the pre-existing general population of users to provide insight that could help the team guide prospective and case studies for cancer survivors in cardio-oncology.
2.5. Innovation technology users diversity
We initiated retrospective study data analysis regarding pre-existing users of the digital health tools. Companies offered technologies/software that were “patient-facing,” “clinician-facing,” or both “patient- and clinician-facing,”; the majority of the interactions with an interface or technology were presented to and pursued by one or both of these types of parties. The term “clinician-facing” encompassed all users that provide care for patients, including physicians, nurses, pharmacists, therapists, and other health care professionals. The majority of the companies offered “patient-facing” products; individuals used the product to benefit their own health. Notably, this term has generally been used to describe individuals who are instructed to use the product for medical purposes. For instance, patient users of innovative technologies have included children and adult patients who have cancer, diabetes, CVD, and other chronic conditions. More broadly, innovative technologies can benefit anyone who has cardiovascular health concerns. Beyond individuals who are explicitly considered patients, other users have included people who desire having greater accountability of their health, such as health-conscious non-medical users.
2.6. Innovation accessibility
Website design began, with a subsequent plan for a mobile application for C.H.I.R.P. to direct patients and providers to resources and technologies. The website was hosted on Wix.com, a website development company. The mobile application was also designed using Spaces by Wix. These interfaces provided summaries of our retrospective and prospective studies in patient-friendly language, recommendations on which patient populations would reap the most benefits from each software/technology, and details on how patients could access the technologies/software. The website and mobile application consolidated an extensive body of information into resources that could be easily navigated by patients and clinicians alike. Having these accessible resources would be especially meaningful as our program would continue to expand and partner with a growing number of digital health companies. The website located at https://brownvirtuallab.wixsite.com/chirp, and would eventually be hosted at https://www.cardioonccoin.org/chirp.
2.7. Electronic health records integration
We partnered with companies providing uniform, common interfaces for EHR and remote technologies such as wearables, consumer home medical devices, and implanted cardiac rhythm devices, to enhance connectedness of health data and efficiency in clinical outcome management. EHR integration was discussed and would require software engineers to build application programming interfaces (APIs) between multiple organizations. Each EHR would have an API that external software providers could utilize to send data. The external software matched fields in their database to the fields in the EHR via a mapping process that then would send data back and forth, or in one direction. Our partner companies utilized Health Level Seven (HL7) & Fast Healthcare Interoperability Resources (FHIR) APIs to integrate with EPIC and eClinicalWorks. We crafted a plan accordingly, for similar integration, with anticipated incorporation of physiological monitoring data.
2.8. Innovation program building and education
Our program building also involved outreach to individuals in a network-affiliated hospital and health systems, as well as people around the country and world. We performed this outreach to gauge people's interest in C.H.I.R.P., the directions they want to pursue with our program, and best practices they would suggest when building the program. Events of collaboration and trainee presentation were targeted as learning opportunities for C.H.I.R.P. members, ways to receive feedback and ways to recruit partners and collaborators. Educational opportunities have been pursued through collaboration with the Cardiology Oncology Innovation Network (COIN; CardioOncCOIN.Org). COIN is a larger innovation network which consists of individuals in cardiology, oncology, and other branches of health care, industry, and finance. COIN is managed by the parent company of Preventive Cardio-Oncology LLC (PrevCardioOnc.Com). We have participated in several events hosted by COIN such as quarterly meetings and yearly national summits that have afforded our group the opportunity to educate others on C.H.I.R.P. and to learn about similar projects at other institutions. Educational opportunities for C.H.I.R.P. members have included abstracts, poster presentations and breakout room discussions about the role of digital health in cardiology and oncology.
Within the network of C.H.I.R.P. partner companies, one company contributed to program building efforts by hosting patient education related to C.H.I.R.P. as well as affiliated digital health companies on a unified platform. Through this platform, patients and clinicians interested in C.H.I.R.P. could view instructional videos and modules that explained the purpose of C.H.I.R.P. and the innovative technologies/software offered by each company. Hosting C.H.I.R.P. patient education through this platform served as a useful complement to the mobile application and website being created to enhance patient and clinician engagement.
3. Results
3.1. Innovation company screening and recruitment
From introductory meetings with company leaders, we established working partnerships to discuss features of their technologies and the feasibility of conducting research studies assessing adoption and clinical efficacy of their technology or software. Not every explored partnership materialized. C.H.I.R.P. screened ~30 companies through the AMA innovation network and LinkedIn, recruited ~20, and collaborated with ~10 (Fig. 1). Partnership retention was built on continuous, transparent communication with companies, standing meetings with researchers and company representatives, and succinct study proposals to serve as guides in answering research questions.
In Table 1 are outlined the specific technology of each collaborator, the number of estimated deidentified users in retrospective studies, salient features of the data they collected, and the intended clinical impact of these technologies. These companies focused on a broad array of digital health applications, including: smart shirt with ECG leads and AI and machine learning based diagnostic software (N = 200 users); ECG/BP patterns tracked with smart watch with longitudinal tracking integrated into virtual care and EHR (N = 180); curated audio motivation, mindfulness, self-care, and exercise sessions for individuals with cancer (N = 5000); a metabolism tracking app and device incorporating nutritionist coaching and AI glucose monitoring (N = 2000); a virtual care platform with EHR and remote technology integration (N = 500); behavior modeling video modules (N = 700); vitals monitoring with live measurements and alerts (N = 200); and a biometric tracking app to assess wellness, fatigue and overall health (N = 400).
Table 1.
Company | Tech/number of users (N) | Features | Impact |
---|---|---|---|
Company A | ECG smart shirt/ N = 200 |
Machine learning, AI-based diagnostics; measuring heart rhythm, HR, stress, fatigue levels | Increased management of cardiovascular health |
Company B | ECG/BP smart watch/ N = 180 |
ECG/BP patterns, longitudinal tracking integrated into virtual care and EHR | Increased management of cardiovascular health |
Company C | Audio companion app/N = 5000 | Curated sessions for motivation, mindfulness, self-care, exercise | Improved wellness of cancer patients |
Company D | Metabolism tracking app/ N = 2000 |
Metabolism tracking with nutritionist and coaching services, integrating AI for glucose level monitoring | Glucose level management |
Company E | Virtual care platform/ N = 500 |
Integration with EHR and remote technologies | Improve healthcare delivery with digital health data |
Company F | Patient education modules/ N = 700 |
Instructional modules modeling patient behavior and health practices | Patient literacy and improving health outcomes and reduce readmission |
Company G | BP monitor/ N = 200 |
Live-time measurements and automated data storage, cardiac alerts, pulse oximetry, HR | Increased management of cardiovascular health |
Company H | Biometrics tracking app/ N = 400 |
Track and assess wellness, fatigue and overall health | Improve sleep, mood, stress management |
AI: Artificial Intelligence; BP: Blood Pressure; ECG: Electrocardiography; EHR: Electronic Health Record and HR: Heart Rate.
3.2. Innovation research team and foundation
We formed an interdisciplinary and multi-level research team to reap the benefits of various perspectives and skillsets of our team members. The cardiology and oncology attending physicians and fellows, medical residents, medical and graduate students, and undergraduate and high school students that make up the C.H.I.R.P. team have filled different roles within C.H.I.R.P. and have directly collaborated with innovators and entrepreneurs in the digital health space (Fig. 5). The distribution of tasks outlined in Fig. 5 have allowed for efficient delegation of tasks with respect to the time commitments and levels of training of our team members. Beyond the tasks of study design, manuscript preparation, and literature reviews, medical and graduate student trainees have had the opportunity to act as liaisons between resident and attending physicians and digital health companies. This valuable experience has empowered students with inter-professional competencies including corresponding and designing studies with professionals in greater healthcare research and collaboration.
Through the partnerships established with digital health companies, deidentified retrospective data were obtained and retrospective studies were initiated, focused on adoption, user satisfaction, and clinical efficacy for various digital health technologies in C.H.I.R.P.. In these retrospective analyses, evaluation of how these technologies promote health in the general population began, with the goal of ultimately assessing how these technologies promote health in cancer survivors. An accessible suite of technologies was created for patient care, engagement, education, and outcomes, to optimize general health, wellness, and cardiovascular health, especially in the cardio-oncology population.
3.3. Innovation technology users diversity
We partnered with one company providing a primarily clinician-facing product and six companies providing primarily patient-facing products. Two companies provided products that are both patient- and clinician-facing. Importantly, data from patient-facing technologies, which were mainly wearables and mobile applications, could be integrated into our clinician-facing technologies to enhance care delivery.
3.4. Innovation accessibility
On the new website (https://www.cardioonccoin.org/chirp), information on each of collaborating companies was presented in a patient-friendly manner. Healthcare providers were able to guide patients to this website, for descriptions of digital health interventions appropriate for the patients' conditions. In addition, the patient-facing webpage provided information about each C.H.I.R.P. company, with summaries of each technology and guidance for using these technologies. The patient access page additionally pointed to each company's own website, with illustrative videos and patient education materials. The clinician-facing webpage pointed to sections of the company pages describing results from pilot studies and clinical trials previously completed with other groups. The clinician access page was also planned to eventually summarize results from retrospective, prospective, case studies completed within or affiliated with C.H.I.R.P.. A mobile application was also used as an option for accessing information from the website.
3.5. Electronic health records integration
Two of our partner companies created a platform integrating electronic health records with health data uploaded from wearables and other devices. Data was contextualized with EHR information so that changes in our patients' health were readily monitored. For example, if an abnormal heart rhythm is recorded for a patient with a history of arrhythmias, the company and client clinician teams managing the platform could quickly message the patient with follow-up concerns. The platforms allowed for two-way communication between patients and clinicians, so patients can rapidly respond to their providers' concerns regarding their health, and vice versa. EHR integration further functioned to reduce administrative time for clinical staff that could be otherwise spent caring for patients, and integration reduced data disorganization between various EHR systems keeping track of health information.
3.6. Innovation program building and education
C.H.I.R.P was fundamentally affiliated with C.O.I.N, a larger innovation network. By working with partners across the larger network, we expanded our reach, shared our capabilities to study different technologies, and provided collaborative information to cardiology and oncology clinics and programs and their respective patient populations across the country. Our group hosted quarterly meetings with COIN to disseminate our study and program design and share suggestions with other professionals within the network that had unique perspectives and expertise and also planned to integrate digital health in cardio-oncology. A roadmap was charted for those within the COIN network hoping to initiate digital health focused research projects at their institutions (Fig. 4). A COIN member obtained an IRB protocol approved at their institution for analysis of digital health products, and another member at a different institution subsequently set up a prospective study with a partnered digital health company through C.H.I.R.P..
In December 2021, our team moderated discussion during the COIN second annual virtual summit [23] to highlight the importance of clinician education in innovation in cardiology and oncology and especially at the interface in cardio-oncology, in our case with a focus on C.H.I.R.P.. The discussion fostered the inclusion of the following elements into C.H.I.R.P. program building:
-
•
Enhancement of the patient representative contributors to the planning and process of C.H.I.R.P. research,
-
•
Incorporation of primary care providers (PCPs) in the C.H.I.R.P network; PCPs should be aware of cardiotoxicities and longitudinal care in patients that receive cancer therapies, with cardio-oncology and digital health subspecialty considerations in primary care for these patients,
-
•
Expansion of C.H.I.R.P. to reach as many health care professionals caring for individuals with cancer as possible (beyond cardiology and primary care), given that patients wish to start cancer therapy immediately after receiving a cancer diagnosis from their hematologists/oncologists and post-cancer cardiotoxicities may be a secondary consideration at that time, prior to direct interaction with cardio-oncology.
In addition to the moderated breakout room discussion session, trainees presented a poster on “Building C.H.I.R.P.” (Fig. 6) at the COIN 2021 Summit (see poster abstract in COIN 2021 summit booklet, https://www.cardioonccoin.org/2021-attendee-booklet; or in abstract book published on Frontiers. An academic oncologist and past chair of the American Society of Clinical Oncology Cancer Survivorship Committee moderated the overall trainee abstract session. Following each virtual presentation, abstract presenters received feedback from a group of over 30 professionals and society leaders in cardiology and oncology, as well as industry. This international virtual presentation increased visibility of the newly established C.H.I.R.P. program and created further opportunity for collaboration throughout COIN.
The specialty and leadership audience questions queried 1) whether patient representatives were incorporated into C.H.I.R.P., and 2) whether connected and digital health would be integrated into EHRs. The first question helped guide discussions on the enhanced integration of patient perspectives in C.H.I.R.P., while the second question was answered with references to Substitutable Medical Applications and Reusable Technologies on Fast Healthcare Interoperability Resources (SMART on FHIR). SMART is an application platform for EHRs that is based on standards and ensures interoperability. FHIR is a set of standards to help ensure that interoperability. With SMART on FHIR, various remote monitoring components can be integrated into the EHR for unified viewing and patient care [24]. Related topics such as dashboards to view digital health data and “Genius Bars” to help patients get started with their new digital health technologies were also discussed. In summary, these questions from field experts raised important considerations and provided an opportunity for the C.H.I.R.P. team to share specifics of the program with other institutional, industry, and cardiology and oncology society leaders.
Beyond the experience that trainees have gained while presenting at the COIN 2021 Summit, C.H.I.R.P. has also empowered medical and graduate students involved in the program design with a unique educational opportunity. Students involved in conducting retrospective and prospective analyses acquired Collaborative Institutional Training Initiative (CITI) training certification, specifically in Biomedical Research and Good Clinical Practices. These students gained leadership skills by mentoring their peers on communicating and coordinating studies with industry and facilitating working sessions with collaborators to guide data analysis. C.H.I.R.P. students attended manuscript writing and literature search instructional talks and used these skills to assist each other with manuscript production. The educational opportunities that C.H.I.R.P. offered continued to grow as the program expanded and innovative ideas were continually incorporated.
4. Discussion
We established partnerships and written proposals for initial studies with several digital health companies that focus on biometrics tracking, EHR integration, and educational videos and audio applicable to cardio-oncology (Table 1). We formed an interdisciplinary research team consisting of medical and graduate students, physicians, scientists, and individuals engaged in digital health (Fig. 3B). A foundation was built for future retrospective, prospective, and case studies on the utility and feasibility of the applications of digital health software and technologies, for improving clinical outcomes in the general population, and in cardio-oncology patients (Fig. 3C). Innovative digital health technologies were either clinician-facing, patient-facing, or a combination of both, leading to a diverse group of users. A program website was built to connect patients with the tools offered by digital health companies and to facilitate clinician access to pertinent information on these companies and their technologies. Most digital health companies would eventually need to rely on EHR integration to some degree; C.H.I.R.P. worked closely with companies focused on this anticipated integration.
4.1. Innovation company screening and recruitment
C.H.I.R.P. developed working relationships with several industry digital companies and initiated research studies. Analysis of complex and broad digital health data, including millions of continuous glucose monitoring entries, past medical history, and lifestyle health metrics of users. C.H.I.R.P. relied on companies initiating collaboration and also directly contacts digital health companies to establish research relationships. Following initial contact, the research team arranged to meet virtually with interested industry partners to align goals for collaboration, a key step in successful academic-industry collaboration [25]. Partnerships forged with companies with stored user data, a willingness to collaborate, and cardio-oncology applicable technology were most successful. The quantity, format, and usability of data were found central to conducting research [26].
4.2. Innovation research team
Team science was considered vital to addressing complex problems of healthcare, including digital health integration and assessment [27]. Peer mentorship within research teams has been shown to be a valuable form of learning for graduate students [28], and facilitated future career success in medical research [29]. Therefore, building a multi-career stage [30] and interdisciplinary research team added senior and peer mentorship to C.H.I.R.P. Having this diverse research team created an environment in which undergraduate students, medical students, residents, and fellows could learn from each other and from attending physicians (Fig. 5). The lead attending physician, a cardiologist, worked closely with oncologists, internal medicine physicians, and a multitude of executives and representatives with varying specializations in the health business and technology world. Our research team often sought the input of specialists from many fields, in collaboration on a single product. Specialists included electrophysiologists, oncologists, radiologists, and several others providing their input and clinical and industry experiences. Having a diverse group of researchers provided many benefits, including collective thinking that leveraged each team member's network and optimizing creativity [27]. Interdisciplinary research in digital health indeed resulted in higher quality research products [31]. Having different perspectives from various specialists and those outside of clinical medicine created a wider reaching and more accessible result, while maintaining the close scrutiny of physician scientists.
Thus, C.H.I.R.P. operated as an integrated research team, with research collaboration and investigator-initiated research [32]. The team members had shared goals and had built trust through social activities during and outside of lab time. We strove to follow the characteristics of an effective team including: effective leadership and management skills, self-awareness and other-awareness, strategies developed for communicating openly, setting shared expectations and defining roles and responsibilities, creating, sharing, and revisiting a shared vision, making provisions for appropriate recognition and credit, promoting disagreement while containing conflict, learning each other's languages, and enjoying the science and the work together [32].
4.3. Innovation research foundation
An increase in university-industry publications over the last several years was noted [33]. The differing skillsets of academia and industry complemented each other, creating an optimal environment for collaboratively assessing technology [33]. Our collaborating companies focused on an array of digital health technologies (Table 1). Our retrospective, prospective, and case studies in C.H.I.R.P. were designed to study the impact of rapid deployment and adoption of these technologies in the general population, for translation to cardio-oncology patients as applicable in IRB-approved studies.
4.4. Innovation technology users diversity
It was important to recognize that when assessing digital health technologies, a diverse cohort of users could be encountered, mainly: the general population, patients, and clinicians [34,35]. Within these groups, there were further subgroups, including general population early adopters or “super users” who avidly tracked their biometrics to maximize their health or optimally manage their health condition(s). Patients often used patient-facing technologies to understand and monitor their condition, and clinicians used clinician-facing digital health to optimize patient monitoring, outcomes, and clinical efficacy and efficiency [36,37].
4.5. Innovation accessibility
Technology literacy could limit the accessibility of digital health [38]. For those who were not technologically inclined, C.H.I.R.P. hoped to improve accessibility by making it easier for patients to find and assess digital health products in one place. There were several other digital health accessibility considerations. Few digital health features were developed with the 15 % of the world's population with impairments in mind. Using universal design to create products that were effective for as many people as possible would be an essential way to ensure health equity within digital health [39]. Relatively little is known about how to improve digital health accessibility from a cultural standpoint [40], which is pivotal in a country as diverse as the United States. Digital health tools and studies in programs such as C.H.I.R.P. would need to remain culturally accessible and available to all.
4.6. Electronic health records and health system integration
Planning for incorporation of digital health data into existing EHR technologies was considered imperative for digital health integration into the healthcare system, but faced many challenges that together we began to explore in order to advance the field [41]. For EHR integration, C.H.I.R.P. planned to incorporate remote wearable and monitoring digital health data into patients' charts directly through a unified common interface. This would bridge the gap between remote monitoring and in-clinic visits, providing a holistic view of a patient's health. We began collaborating with a start-up company focused on integrating remote patient monitoring and wearable physiological pattern information into patient charts, through an existing cardiology focused virtual care platform. Many health organizations were working on similar projects and insurance companies were even encouraging the use of wearables for subscribers [42]. As wearables would become more pervasive, efficient melding with existing EHR platforms using platforms like SMART on FHIR would allow clinicians to access health information from outside of the typical controlled clinical setting [43] and provide care that acts on data not previously available [24].
4.7. Innovation program building
Research networks have become increasingly relevant in science [44] and we are leveraging this shift through the Cardiology Oncology Innovation Network (COIN). COIN membership consists of clinicians associated with over 50 academic hospitals/ health systems and C.H.I.R.P. is a concrete step in the tripartite COIN mission of innovation, collaboration, and education [45]. Our team's collaborations through this network already initially shaped our program, with the enhanced incorporation of patient representatives and development of “Genius Bars” to help patients get started with their digital health products, based on feedback during the COIN 2021 Summit. Patient representative feedback ensured that digital health research and implementation, through C.H.I.R.P., would be optimized for patient experience and emphasize patient-centered care. COIN members also took active roles in manuscript publications.
4.8. Innovation education
Medical students, graduate students, and other students were provided an opportunity to learn more about the research pipeline in academia from developing research study design, working with regulatory IRB committees, enhancing skills in scientific inquiry, and writing medical literature (Fig. 5). With a longitudinal, multifaceted program such as C.H.I.R.P., students were exposed to tremendous opportunities to supplement their academic education with valuable exposure to clinical and translational research [46]. Academia would often exclude racial minorities and research teams would frequently lack diversity. The Brown Lab PI actively worked to improve these disparities through student recruitment and by focusing on these issues in research products. Medical students who identified with a racial minority group would often cite lack of senior role models similar to them and less opportunities and inclusivity in academic pursuits as reasons to not engage is academia [47]. By placing inclusion and improvement of the current state of academia at the forefront, the Brown Lab and C.H.I.R.P. produce higher quality research that was equitable and paved the way for similar research teams.
4.9. Team science challenges, limitations, strengths
Team science has been transforming academic research for decades and has helped shift production of knowledge away from a model of individual genius, to a combined effort between and within institutions [48].The leading C.H.I.R.P. physician has experience forming diverse research teams and networks that fit within this team science model [49,50]. Obstacles can be common in building complex, interdisciplinary team science research programs such as C.H.I.R.P. [51]. Selecting particular paths with precision based on available personnel can alleviate many of these challenges and facilitate expanding the availability of relevant technologies for patients.
5. Conclusion
Despite the growing need for improved monitoring, surveillance, and use of tools for cardiovascular health and wellness in the general population, and especially in cancer survivors, this remains an untapped area of research. With cardio-oncology as an emerging field due to the cardiovascular complications that arise from cancer therapies, connected health platforms such as C.H.I.R.P. address this need through innovation and patient, clinician, and trainee education, with a goal of improving care delivery systems through digital health. Barriers to digital health access often limit diversity of digital health users, making user samples unrepresentative of the general population [52,53]. We have acknowledged this limitation in our retrospective manuscripts and are planning future studies that will decrease cost and accessibility barriers to be more representative of the general population. Researchers can similarly take steps to combat these barriers to access by noting diversity limitations in user populations, and by advocating for underserved communities gaining access to these technologies. Users may have varying motivations for using digital health products, and care should be taken to incorporate and analyze the user population in study designs with each digital health company. To capture diverse user motivation, we have incorporated patient representatives to ensure a patient-centered approach. Utilizing technology to improve access to health and wellness monitoring and facilitate engagement with digital tools will hopefully lead to improved quality of life and reduced costs for monitoring and maintaining cardiovascular health in the general population and in cardio-oncology.
Funding
This publication was supported by the National Center for Advancing Translational Sciences, National Institutes of Health, through Grant Numbers UL1TR001436 and KL2TR001438. Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the NIH. This work was also supported by the American Heart Association-Strategically Focused Research Network Grant in Disparities in Cardio-Oncology (#847740, #863620).
CRediT authorship contribution statement
Conception and design: SAB
Drafting of the manuscript: SM, JM, TM, SP, AS, AR, AH, SAB, AB
Interpretation of data: SM, JM, TM, SP, AS, AR, IC, AH, SAB
Critical revision: SM, JM, TM, SP, AS, AH, SAB, AB, MA, HM, AG, BP, DA
Final approval of manuscript: All authors
Declaration of competing interest
The authors declare the following financial interests/personal relationships which may be considered as potential competing interests:
All of our authors work closely with several health technology companies, none of whom inappropriately restrict or limit our analyses or publications. Dr. Brown serves as an advisor for two of the companies and has not been paid for any work related to this manuscript.
Acknowledgements
We are grateful to all members of the Connected Health Innovation Research Program (C.H.I.R.P.), Cardiology Oncology Innovation Network (COIN), and Cardio-Oncology Artificial Intelligence Informatics & Precision (C.A.I.P.) Research Team Investigators for local, regional, national and global partnership in this work.
References
- 1.Yusuf S., Hawken S., Ounpuu S., Dans T., Avezum A., Lanas F., et al. Effect of potentially modifiable risk factors associated with myocardial infarction in 52 countries (the INTERHEART study): case-control study. Lancet. 2004;364(9438):937–952. doi: 10.1016/S0140-6736(04)17018-9. [DOI] [PubMed] [Google Scholar]
- 2.Siegel R.L., Miller K.D., Jemal A. Cancer statistics, 2020. CA Cancer J. Clin. 2020;70(1):7–30. doi: 10.3322/caac.21590. [DOI] [PubMed] [Google Scholar]
- 3.Mehta L.S., Watson K.E., Barac A., Beckie T.M., Bittner V., Cruz-Flores S., et al. Cardiovascular disease and breast cancer: where these entities intersect: a scientific statement from the American Heart Association. Circulation. 2018;137(8):e30–e66. doi: 10.1161/CIR.0000000000000556. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Dent S.F., Kikuchi R., Kondapalli L., Ismail-Khan R., Brezden-Masley C., Barac A., et al. Optimizing cardiovascular health in patients with cancer: a practical review of risk assessment, monitoring, and prevention of cancer treatment-related cardiovascular toxicity. Am. Soc. Clin. Oncol. Educ. Book. 2020;40:1–15. doi: 10.1200/EDBK_286019. [DOI] [PubMed] [Google Scholar]
- 5.Iacopo F., Branch M., Cardinale D., Middeldorp M., Sanders P., Cohen J.B., et al. Preventive cardio-oncology: cardiovascular disease prevention in cancer patients and survivors. Curr. Treat. Opt. Cardiovasc. Med. 2021;23(1):8. [Google Scholar]
- 6.Martinez H.R., Beasley G.S., Goldberg J.F., Absi M., Ryan K.A., Guerrier K., et al. Pediatric cardio-oncology medicine: a new approach in cardiovascular care. Children (Basel) 2021;8(12) doi: 10.3390/children8121200. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Ruddy K.J., Sangaralingham L.R., Van Houten H., Nowsheen S., Sandhu N., Moslehi J., et al. Utilization of cardiac surveillance tests in survivors of breast cancer and lymphoma after anthracycline-based chemotherapy. Circ. Cardiovasc. Qual Outcome. 2020;13(3) doi: 10.1161/CIRCOUTCOMES.119.005984. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Koop Y., El Messaoudi S., Vermeulen H., Maas A.H.E.M., Atsma F. Healthcare utilization and hospital variation in cardiac surveillance during breast cancer treatment: a nationwide prospective study in 5000 Dutch breast cancer patients. Cardiooncology. 2020;6:14. doi: 10.1186/s40959-020-00068-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.de Boer R.A., Aboumsallem J.P., Bracun V., Leedy D., Cheng R., Patel S., et al. A new classification of cardio-oncology syndromes. Cardiooncology. 2021;7(1):24. doi: 10.1186/s40959-021-00110-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Okwuosa T.M., Ray R.M., Palomo A., Foraker R.E., Johnson L., Paskett E.D., et al. Pre-diagnosis exercise and cardiovascular events in primary breast cancer. Women’s Health Initiative. 2019;1(1):41–50. doi: 10.1016/j.jaccao.2019.08.014. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Kagiyama N., Shrestha S., Farjo P.D., Sengupta P.P. Artificial intelligence: practical primer for clinical research in cardiovascular disease. J. Am. Heart Assoc. 2019;8(17) doi: 10.1161/JAHA.119.012788. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Duke MEDx Available from. 2020. http://medx.duke.edu/about
- 13.Best Practices for Consumer Cardiovascular Technology Solutions . 2022. American College of Cardiology: Consumer Technology Association. [Google Scholar]
- 14.Digital Health Technologies for Remote Data Acquisition in Clinical Investigations . 2022. FDA Guidance Documents: Food and Drug Administration. [Google Scholar]
- 15.Gabizon I., Bhagirath V., Lokker C., Bhavnani S.P., Lonn E. What do physicians need to know in order to ‘prescribe’ mobile applications to patients with cardiovascular disease? Perinat. Med. 2019;16(4):263–268. doi: 10.2217/pme-2019-0015. [DOI] [PubMed] [Google Scholar]
- 16.Lim W.K., Davila S., Teo J.X., Yang C., Pua C.J., Blöcker C., et al. Beyond fitness tracking: the use of consumer-grade wearable data from normal volunteers in cardiovascular and lipidomics research. PLoS Biol. 2018;16(2) doi: 10.1371/journal.pbio.2004285. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Maddox T.M., Albert N.M., Borden W.B., Curtis L.H., Ferguson T.B., Kao D.P., et al. The learning healthcare system and cardiovascular care: a scientific statement from the American Heart Association. Circulation. 2017;135(14) doi: 10.1161/CIR.0000000000000480. (e826-e57) [DOI] [PubMed] [Google Scholar]
- 18.Foraker R.E., Benziger C.P., DeBarmore B.M., Cené C.W., Loustalot F., Khan Y., et al. Achieving optimal population cardiovascular health requires an interdisciplinary team and a learning healthcare system: a scientific statement from the American Heart Association. Circulation. 2021;143(2):e9–e18. doi: 10.1161/CIR.0000000000000913. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Leu F.Y., Ko C.Y., You I., Choo K.-K.R., Ho C.-L. A smartphone-based wearable sensors for monitoring real-time physiological data. Comput. Electr. Eng. 2017;65:376–392. [Google Scholar]
- 20.Peake J.M., Kerr G., Sullivan J.P. A critical review of consumer wearables, mobile applications, and equipment for providing biofeedback, monitoring stress, and sleep in physically active populations. Front. Physiol. 2018;9:743. doi: 10.3389/fphys.2018.00743. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Bhavnani S.P., Harzand A. From false-positives to technological Darwinism: controversies in digital health. Perinat. Med. 2018;15(4):247–250. doi: 10.2217/pme-2018-0033. [DOI] [PubMed] [Google Scholar]
- 22.Bhavnani S.P., Parakh K., Atreja A., Druz R., Graham G.N., Hayek S.S., et al. 2017 roadmap for innovation-ACC health policy statement on healthcare transformation in the era of digital health, big data, and precision health: a report of the American College of Cardiology Task Force on health policy statements and systems of care. J. Am. Coll. Cardiol. 2017;70(21):2696–2718. doi: 10.1016/j.jacc.2017.10.018. [DOI] [PubMed] [Google Scholar]
- 23.Brown S.A., Beavers C., Bauer B., Cheng R.K., Berman G., Marshall C.H., et al. Advancing the care of individuals with cancer through innovation & technology: proceedings from the cardiology oncology innovation summit 2020 and 2021. Front. Cardiovasc. Med. 2022 doi: 10.1016/j.ahjo.2023.100354. (In revision) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Mandel J.C., Kreda D.A., Mandl K.D., Kohane I.S., Ramoni R.B. SMART on FHIR: a standards-based, interoperable apps platform for electronic health records. J. Am. Med. Inform. Assoc. 2016;23(5):899–908. doi: 10.1093/jamia/ocv189. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Vedel J.B., Irwin A. ‘This is what we got, what would you like?’: aligning and unaligning academic-industry relations. Soc. Stud. Sci. 2017;47(3):417–438. doi: 10.1177/0306312716689346. [DOI] [PubMed] [Google Scholar]
- 26.Jain S.H., Rosenblatt M., Duke J. Is big data the new frontier for academic-industry collaboration? JAMA. 2014;311(21):2171–2172. doi: 10.1001/jama.2014.1845. [DOI] [PubMed] [Google Scholar]
- 27.Disis Mary L, Slattery John T. The road we must take: multidisciplinary team science. Sci. Transl. Med. 2010;2(22):22cm9-cm9. [DOI] [PubMed]
- 28.Meschitti V. Can peer learning support doctoral education? Evidence from an ethnography of a research team. Stud. High. Educ. 2019;44(7):1209–1221. [Google Scholar]
- 29.Begg M.D., Crumley G., Fair A.M., Martina C.A., McCormack W.T., Merchant C., et al. Approaches to preparing young scholars for careers in interdisciplinary team science. J. Investig. Med. 2014;62(1):14–25. doi: 10.231/JIM.0000000000000021. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Börner K, Contractor N, Falk-Krzesinski HJ, Fiore SM, Hall KL, Keyton J, et al. A multi-level systems perspective for the science of team science. Sci. Transl. Med. 2010;2(49):49cm24. [DOI] [PMC free article] [PubMed]
- 31.Molina Recio G., García-Hernández L., Molina Luque R., Salas-Morera L. The role of interdisciplinary research team in the impact of health apps in health and computer science publications: a systematic review. Biomed. Eng. Online. 2016;15(Suppl. 1):77. doi: 10.1186/s12938-016-0185-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Bennett L.M., Gadlin H. Collaboration and team science. J. Investig. Med. 2012;60(5):768. doi: 10.231/JIM.0b013e318250871d. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.University-Industry Collaboration . 2021. Elsevier. [Google Scholar]
- 34.Sharma A., Harrington R.A., McClellan M.B., Turakhia M.P., Eapen Z.J., Steinhubl S., et al. Using digital health technology to better generate evidence and deliver evidence-based care. J. Am. Coll. Cardiol. 2018;71(23):2680–2690. doi: 10.1016/j.jacc.2018.03.523. [DOI] [PubMed] [Google Scholar]
- 35.Kukafka R. Digital health consumers on the road to the future. J. Med. Internet Res. 2019;21(11):e16359-e. [DOI] [PMC free article] [PubMed]
- 36.AMA Digital Health Research American Medical Association Available from. 2019. https://www.ama-assn.org/system/files/2020-02/ama-digital-health-study.pdf
- 37.Fuller T.E., Pong D.D., Piniella N., Pardo M., Bessa N., Yoon C., et al. Interactive digital health tools to engage patients and caregivers in discharge preparation: implementation study. J. Med. Internet Res. 2020;22(4) doi: 10.2196/15573. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Triana A.J., Gusdorf R.E., Shah K.P., Horst S.N. Technology literacy as a barrier to telehealth during COVID-19. Telemed. J. E Health. 2020;26(9):1118–1119. doi: 10.1089/tmj.2020.0155. [DOI] [PubMed] [Google Scholar]
- 39.Henni S.H., Maurud S., Fuglerud K.S., Moen A. The experiences, needs and barriers of people with impairments related to usability and accessibility of digital health solutions, levels of involvement in the design process and strategies for participatory and universal design: a scoping review. BMC Public Health. 2022;22(1):35. doi: 10.1186/s12889-021-12393-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Marwaha J.S., Kvedar J.C. Cultural adaptation: a framework for addressing an often-overlooked dimension of digital health accessibility. NPJ. Digit Med. 2021;4(1):143. doi: 10.1038/s41746-021-00516-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Shull J.G. Digital health and the state of interoperable electronic health records. JMIR Med. Inform. 2019;7(4) doi: 10.2196/12712. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42.Dinh-Le C., Chuang R., Chokshi S., Mann D. Wearable health technology and electronic health record integration: scoping review and future directions. JMIR Mhealth Uhealth. 2019;7(9) doi: 10.2196/12861. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Tiase V.L., Hull W., McFarland M.M., Sward K.A., Del Fiol G., Staes C., et al. Patient-generated health data and electronic health record integration: a scoping review. JAMIA Open. 2020;3(4):619–627. doi: 10.1093/jamiaopen/ooaa052. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.Adams J. The rise of research networks. Nature. 2012;490(7420):335–336. doi: 10.1038/490335a. [DOI] [PubMed] [Google Scholar]
- 45.Brown S.A., Beavers C., Martinez H.R., Marshall C.H., Olaye I.M., Guha A., et al. Bridging the gap to advance the care of individuals with cancer: collaboration and partnership in the Cardiology Oncology Innovation Network (COIN) Cardiooncology. 2022;8(1):2. doi: 10.1186/s40959-022-00129-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46.Mokshagundam S., Pitkin J., Dekhtyar M., Santen S., Hammoud M., Skochelak S.E. Engaging medical students in leadership development. Med. Sci. Educ. 2019;29(3):849–853. doi: 10.1007/s40670-019-00754-w. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Curtis-Lopez C., Robinson D., Shirke M., Dominic C., Patel R. Drivers and barriers to engaging with academia: a minority-ethnic medical student perspective. J. R. Soc. Med. 2021;114(10):470–472. doi: 10.1177/01410768211029156. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 48.Wuchty S., Jones B.F., Uzzi B. The increasing dominance of teams in production of knowledge. Science. 2007;316(5827):1036–1039. doi: 10.1126/science.1136099. [DOI] [PubMed] [Google Scholar]
- 49.Chouvarda I., Mountford N., Trajkovik V., Loncar-Turukalo T., Cusack T. Leveraging interdisciplinary education toward securing the future of connected health research in Europe: qualitative study. J. Med. Internet Res. 2019;21(11) doi: 10.2196/14020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 50.Brown S.-A., Sparapani R., Osinski K., Zhang J., Blessing J., Cheng F., et al. Establishing an interdisciplinary research team for cardio-oncology artificial intelligence informatics precision and health equity. Am. Heart J. Plus Cardiol. Res. Practice. 2022;13 doi: 10.1016/j.ahjo.2022.100094. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 51.DeHart D. Team science: a qualitative study of benefits, challenges, and lessons learned. Soc. Sci. J. 2017;54(4):458–467. [Google Scholar]
- 52.Frederix I., Caiani E.G., Dendale P., Anker S., Bax J., Böhm A., et al. ESC e-Cardiology Working Group Position Paper: overcoming challenges in digital health implementation in cardiovascular medicine. Eur. J. Prev. Cardiol. 2019;26(11):1166–1177. doi: 10.1177/2047487319832394. [DOI] [PubMed] [Google Scholar]
- 53.Whitelaw S., Pellegrini D.M., Mamas M.A., Cowie M., Van Spall H.G.C. Barriers and facilitators of the uptake of digital health technology in cardiovascular care: a systematic scoping review. Eur. Heart J. Digit Health. 2021;2(1):62–74. doi: 10.1093/ehjdh/ztab005. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 54.Chaiyachati K.H., Liao J.M., Weissman G.E., Morgan A.U., Shea J.A., Armstrong K.A. The association between mentor-mentee network features and publication productivity among early career academic generalists. J. Gen. Intern. Med. 2019;34(3):346–348. doi: 10.1007/s11606-018-4702-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 55.Dickson K.S., Glass J.E., Barnett M.L., Graham A.K., Powell B.J., Stadnick N.A. Value of peer mentoring for early career professional, research, and personal development: a case study of implementation scientists. J. Clin. Transl. Sci. 2021;5(1) doi: 10.1017/cts.2021.776. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 56.Nikkar-Esfahani A., Jamjoom A.A.B., Fitzgerald J.E.F. Extracurricular participation in research and audit by medical students: opportunities, obstacles, motivation and outcomes. Med. Teach. 2012;34(5) doi: 10.3109/0142159X.2012.670324. (e317-e24) [DOI] [PubMed] [Google Scholar]