Participatory methods to support team science development for predictive analytics in health

Abstract

Predictive analytics in health is a complex, transdisciplinary field requiring collaboration across diverse scientific and stakeholder groups. Pilot implementation of participatory research to foster team science in predictive analytics through a partnered-symposium and funding competition. In total, 85 stakeholders were engaged across diverse translational domains, with a significant increase in perceived importance of early inclusion of patients and communities in research. Participatory research approaches may be an effective model for engaging broad stakeholders in predictive analytics.

Introduction

Predictive analytics in health is an emerging transdisciplinary field utilizing techniques from computer science (e.g., machine learning, signal processing), statistics, clinical medicine, and social and behavioral sciences to predict individual and group-level health outcomes [1, 2]. It involves the collection, merging, and analysis of multiple types of individual-level data such as electronic health records, publicly-available administrative data, mobile phone activity, and increasingly, passive sensing technologies [3]. With these advances, concerns have emerged regarding data privacy, ownership, and the risk/benefits related to predicting sensitive, individual-level health outcomes and behaviors [4]. This is especially relevant in low-income, racial/ethnic minority populations that often have limited trust in and access to research opportunities [5, 6]. Team science approaches prioritizing broad stakeholder engagement in research leadership may facilitate advances in predictive analytics by integrating knowledge and priorities across disciplines and perspectives.

As noted by the Institute of Medicine [7] and the National Institutes of Health [8], team science is a collaborative, transdisciplinary approach recommended for accelerating development of clinical translational science into public health impact. Team science addresses complex, multi-faceted questions by including scientists from diverse disciplines, and in some models, the end-users of biomedical research (e.g., community members, patients, clinicians, healthcare systems, payers). To foster effective team science, the Institute of Medicine report included several key recommendations including supporting high diversity of membership, shared vocabularies, and deep knowledge integration. Although previous reports have described factors associated with effective translational team science broadly [9, 10], little is known about how to encourage team science in the emerging field of predictive analytics in health. Community-partnered participatory research (CPPR), a variant of community-based participatory research [11], was developed to support effective partnerships between researchers and stakeholders from under-resourced communities, may be one approach to foster team science in this field [6]. CPPR focuses on early engagement of stakeholders in research co-leadership. The goal is to build trust, engagement and a shared culture across stakeholders through equal power sharing, identification of common interests, and ongoing workgroups [12]. These participatory techniques have been used successfully in bridging cultural gaps to form partnerships between stakeholders from under-resourced communities and academic institutions [6, 11], with increasing examples applied to health informatics [13]. However, it is unclear how these methods may help foster partnerships across the disparate academic disciplines, translational domains, and institutions relevant to predictive analytics.

In this report, we pilot a Clinical Translational Science Award (CTSA)-funded engagement approach to support predictive analytic research that both implements partnered research methods in organizing a team science symposium and introduces these principles to participants through group learning. We primarily focused on addressing the challenge of building partnerships between stakeholders across diverse academic disciplines, translational domains and institutions. We hypothesized that this would support engagement of translational researchers in predictive analytics across disciplines and increase participants’ perceived importance of stakeholder inclusion in research co-leadership.

Methods

Participatory Engagement

We implemented participatory research (CPPR) best practices to stimulate transdisciplinary planning for a scientific symposium on translational team science in predictive health analytics. This included early and diverse stakeholder inclusion, shared decision-making, and building capacity for partnered planning and implementation of research-informed programs [7, 13]. This scientific development initiative and evaluation were funded through a University of California Los Angeles (UCLA) Clinical and Translational Science Institute (CTSI) Catalyst Award. We applied partnered techniques to engage diverse academic disciplines and institutions in planning and participating in a symposium and pilot award program. All planning was conducted through a stakeholder advisory panel with members from 3 CTSA institutions including University of California Los Angeles, University of Southern California, and University of Pittsburgh. The UCLA CTSI provided pilot funding for this project. Stakeholders represented diverse disciplines relevant to predictive health analytics (e.g., statistics, informatics, computer science, clinical medicine, health services research, social sciences, healthcare providers, and administrators).

Symposium and Pilot Competition

The partnered approach to planning through the stakeholder advisory board over a 2-month period resulted in development of 2 structured activities: (1) a day-long symposium and (2) a pilot competition, to encourage new transdisciplinary, translational team science around predictive analytics in health. The symposium and pilot competition were held in December 2013. Additionally, 3-year outcomes for the pilot competitions were obtained in December 2016. During pre-symposium planning, the advisory board identified three themes for the symposium including opportunities for utilizing health system data, predictive analytic methods, and partnered engagement strategies. Three keynote talks were organized to address these themes: (1) reduction of hospital readmission after acute cardiac events using predictive analytics of electronic health record services data, (2) engineering approaches to behavioral signal processing including use of mobile sensors, and (3) an overview of CPPR and implications for predictive analytics.

Advisory board members identified individuals and relevant email distribution lists (e.g., departmental or interest group) to send invitations for prospective attendees. The symposium consisted of an in-person meeting in Long Beach, CA with simultaneous, online participation from Pittsburgh, PA, San Francisco, CA and a keynote address from Dallas, TX. To encourage participation and co-learning, the symposium started with an opportunity for in-person participants to present one slide on their work or interests and remote attendees to introduce their work via videoconference. Thirty-seven of the in-person attendees presented a slide (52%) and spoke approximately 1–3 minutes each. Examples include a summary of interests (e.g., analytic, methodological, or content expertise), a key figure or table from recent studies, and conceptual overviews. There were 2 breakout sessions to discuss key questions and topics addressed in the keynote talks. The symposium concluded with a discussion of the pilot competition for 2, $15,000 awards open to teams emerging from the symposium. Pilot awards were prioritized for teams with researchers forming collaborations across disciplines that they had not worked in before and teams with patients, families, and/or community leaders.

Participant Survey

Evaluation surveys (pre-symposium and post-symposium) were distributed as a single packet to attendees at registration with a request to complete the post-symposium survey at the end of the symposium and return to staff before leaving. Pre-symposium surveys consisted of 15 items including organizational affiliations, education, techniques directly used by participants or through collaborations, challenges to work and collaboration, translational domains of their work (e.g., T0–T4) [14], and perspectives on community-partnered methods. Post-symposium surveys consisted of 7 items assessing perspectives related to community-partnered methods and interest in collaborating across disciplines. Of in-person attendees, 63% (n=44/70) completed the pre-symposium survey and 59% (n=40/70) the post-symposium survey. Survey consent was oral and no incentives were offered for participation. We report results of paired, one-tailed t-tests calculated to assess differences between matched survey items, pre-symposium and post-symposium. A p-value of <0.05 was considered statistically significant. The UCLA Institutional Review Board approved of all methods.

Results

The partnered planning approach resulted in 85 individuals (70 in-person, 15 via videoconference) participating in the day-long symposium (Table 1). Most participants reported having an affiliation with an academic institution (86%, n=38 of 44 survey respondents), including across 6 CTSA sites. Thirty-five (80%) reported having only an academic affiliation while the remaining respondents (n=9/44, 20%) reported at least one nonacademic affiliation such as non-profits, start-ups, public agencies, and community organizations. All reported attaining at least a bachelor’s level educational level. Participants primarily from academic institutions were mostly master’s level (n=5/35, 14%), PhD (n=16/35, 46%) and MD (n=6/35, 17%, including those with combined degrees). There were 2 individuals with JD’s and one individual not reporting their educational level. Those participants indicating nonacademic affiliations had similar educational levels (n=1/9 with master’s level, n=8/9 with MD and/or PhD). In total, 77% (n=34/44) indicated the use of techniques or principles across 3 or more approaches. Many reported not having previously used one or more key approaches presented in the symposium. For example, 45% (n=20/44, not in table) did not indicate that they used techniques from predictive analytics. Conversely, of those that had worked in predictive analytics, 58% (n=14/24, not in tables) had not worked in community/patient engagement or social sciences. Participants’ work spanned the translational spectrum from T0 to T4 (Table 2). Participants reported work across multiple translational domains: 75% (n=33/44) reported work in 2 or more translational domains and 30% (n=13/44) in 3 or more domains.

Table 1.

Symposium participants’ baseline characteristics (n=44)

Please select your primary affiliation(s). Select all that apply	n (%)
Academic research institution	38 (86)
Community partner, patient advocate, non-profit organization	7 (16)
Private startup or technology company	3 (7)
My work directly uses techniques or principles from the following areas. Select all that apply	n (%)
	Academic only (n=35)	Other (n=9)
Statistics	25 (71)	7 (78)
Predictive analytics	19 (54)	5 (56)
Genetics	5 (14)	1 (11)
Medical imaging	10 (29)	0 (0)
Biomarkers/physiological	9 (26)	2 (22)
Behavioral/psychological	10 (29)	4 (44)
Community and patient engagement	10 (29)	4 (44)
Social sciences	3 (9)	2 (22)
Health services/implementation	15 (43)	5 (55)
Health quality	10 (29)	4 (44)
Health policy	6 (17)	3 (33)
Clinical care	15 (43)	4 (44)
Engineering	9 (26)	3 (33)
Art and design	6 (17)	3 (33)
Participants indicating use of techniques or principles across≥2 areas	34 (97)	8 (89)
Participants indicating use of techniques or principles across≥3 areas	27 (77)	7 (78)

Table 2.

Distribution of participation across translational research domains (n=44)

I consider my work to be directly involved in the following stages of translational research (select all that apply)	n (%)
T0 is characterized by the identification of opportunities and approaches to health problems	18 (41)
T1 seeks to move basic discovery into a candidate health application	15 (34)
T2 assesses the value of application for health practice leading to the development of evidence-based guidelines	21 (48)
T3 attempts to move evidence-based guidelines into health practice, through delivery, dissemination, and diffusion research	25 (57)
T4 seeks to evaluate the “real world” health outcomes of population health practice	18 (41)
N/A	2 (0)

Key challenges that participants identified related to conducting their work included complexity of data or techniques (61%, n=27/44), perceived relevance to clinical applications (45%, n=20/44) and data ownership/sharing (39%, n=17/44; Table 3). Barriers to collaboration included lack of funding for collaborations (61%, n=27/44), excessive time required (36%, n=16/44) and difficulty identifying relevant collaborators (34%, n=15/44). Eighty-eight percent (n=35/40, not in tables) of respondents agreed or strongly agreed that they found the symposium helpful and planned to integrate lessons learned in future work. The group was more likely after the symposium to agree that it is important to involve patients and community organizations before identifying research questions (pre-symposium: 3.67, post-symposium: 4.15; p<0.05; 1=strongly disagree to 5=strongly agree, not in tables). There was no significant change in agreement related to planning to integrate community-partnered techniques at early stages in future work (pre: 3.77, post: 3.95; p=0.15, not in tables).

Table 3.

Challenges and opportunities identified by stakeholders (n=44)

	n (%)
I feel the most challenging aspect of my work is (select up to 3)
Complexity of the data and computational techniques	27 (61)
Relevance to clinical applications	20 (46)
Data ownership/sharing issues	17 (39)
Adherence by patients	12 (27)
Reproducibility of findings	9 (21)
Connection to biological mechanisms	8 (18)
The main challenges to collaboration are (select up to 3)
Funding	27 (61)
Excessive time/effort required	16 (36)
Identifying individuals in other areas	15 (34)
Lack of perceived need to collaborate by others	13 (30)
Complexity of data/techniques	12 (27)
Data ownership/sharing issues	10 (23)
Distance or other physical barriers	10 (23)
Regulatory/privacy issues	6 (14)

Two pilot grants were awarded as part of the symposium: (1) predicting depression outcomes from electronic health record data and (2) social network analysis to predict autism outcomes. At 3-year follow-up, awardees reported a variety of products resulting from the competition including 8 new pairwise collaborations across 2 CTSA sites, 3 journal publications, and 8 symposium presentations.

Discussion

This report describes a CTSA-funded pilot adapting CPPR methods to foster team science approaches. This included partnered engagement in planning a scientific symposium and pilot funding competition. In addition, we introduced partnered methods during the symposium to develop awareness of the importance of including stakeholders in research leadership. The partnered engagement approach to planning resulted in the broad inclusion of scientists across multiple institutions and diverse translational domains, both with and without experience in predictive analytics. This represents an exposure of participants to novel fields, an important step in facilitating the formation of new team science collaborations. Amongst survey respondents, we observed a significant increase in agreement pre-symposium Versus post-symposium of the importance of involving patients and community organizations early in the research process. However, no significant shift was observed in agreement with planning to integrate community-partnered techniques. Consistent with prior reports from CTSA sites, the participants identified lack of funding as a barrier to collaboration [7]. The symposium pilot funding competition to build participatory team science in predictive analytics represented an opportunity to directly provide this financial support and resulted in multiple journal papers, conference presentations and new collaborations reported by awardees at 3-year follow-up.

Although this study has limited follow-up and small sample sizes, our findings are consistent with a recent report from the Medical University of South Carolina CTSA summarizing 5 years of scientific retreats showing that the scientific symposium format paired with funding opportunities may enhance team science collaborations as measured by co-authored, peer-reviewed publications [15]. However, as team science involves individuals across disciplines and settings, alignment of goals and effective communication across these domains (each with specific knowledge, vocabularies, and scientific approaches) continues to be cited as a challenge [16]. Our results suggest that participatory research approaches may facilitate addressing these challenges through its focus on early and broad stakeholder engagement, equal co-leadership, and support for developing a shared community.

Our approach focused on participatory engagement of primarily academic individuals across disciplines, translational domains and sites. Therefore, there was minimal engagement of patient, family, and community representatives. Inclusion of these broader stakeholders in planning and collaboration efforts would be beneficial and we recommend future efforts include larger symposiums with more involvement of these nonacademic stakeholders. In addition, this pilot was limited to 1-day event. Other groups (including CTSA sites) interested in implementing these partnered methods may consider forming a standing participatory stakeholder advisory board and ongoing activities to support collaborations and build capacity for translational team science over time, paired with longitudinal follow-up to evaluate effectiveness. Comparative studies of this partnered engagement strategy with other approaches to stimulate team science in predictive analytics may also be informative.

Future efforts in predictive health analytics may also explore two challenges identified by participants: data ownership/sharing and enhancing clinical relevance. Updating privacy and ethics guidelines may be needed to address advances in technologies, analytic techniques, and merging of large data sets that may render current methods to protect patient privacy (such data de-identification) obsolete [4, 5, 17, 18]. The relevance of these techniques for predicting individual or group-level outcomes may also be limited by under-sampling of some groups, especially low-income, under-resourced communities, due to well-documented distrust and diminished access to research opportunities. A recent World Economic Forum report describes several related recommendations including the need for transparency, trust, and engagement of broad stakeholders to mitigate risk and to ensure equitable distribution of value resulting from these efforts [19]. Trust and transparency were also cited as important factors to ensure ethical use of technologies in mental health as well [20].

Interdisciplinary collaboration and stakeholder engagement at all research stages is critical to address the unique challenges and opportunities in predictive analytics and related efforts for diverse populations. Inclusion of participatory research approaches in planning and educational activities may be an effective model for engaging broad academic disciplines and stakeholders in advancing predictive analytics in research and healthcare.

Disclosures

A.C.A. is the founder of Open Science Initiative, Arevian Technologies Inc., and Insight Health Systems Inc. S.N. is Chief Scientist at Behavioral Signal Technologies and Behavioral Informatix. All other author have no conflicts of interest to declare.