Community-engaged Evidence Synthesis to Inform Public Health Policy and Clinical Practice

Background:

This case study documents the work of the Rhode Island Arts and Health Advisory Group, which convened in 2016 to develop a set of policy, clinical practice, and research recommendations for implementation by the Rhode Island Department of Health, The Rhode Island State Council on the Arts, and partners. Comprised of artists, clinicians, community members, and patients, the group partnered with researchers to complete an evidence synthesis project of arts-based health care interventions.

Methods:

The group took a community-engaged approach to evidence synthesis, featuring the use of online, and in-person training materials to facilitate the codesign and coexecution of the evidence synthesis protocol. The final evidence map was translated into an online evidence map to facilitate analysis and discussion on arts-based interventions in health care.

Results:

The evidence map informed the development of recommendations for advancing the integration of arts and health in the state. The project evaluation indicated that our community-engaged approach to evidence synthesis promoted engagement as defined by the PCORI Engagement Strategy Rubric (ie, reciprocal relationships, partnership, colearning, transparency, honesty, and trust). Participation also improved community research partners confidence in engaging with the health care system, developed greater empathy and understanding of others in the community, and increased interest in using science or research in advocacy efforts.

Conclusions:

Engaging community partners in evidence synthesis promotes community dialogue and engagement in research, specifically towards: (1) elucidating outcomes of import to patients and communities that are not represented in the medical literature; and (2) identifying comparisons among interventions that resonate with patients and communities.

BACKGROUND

Good decision-making by policy makers, insurers, health systems administrators and funding agencies requires access to evidence that is applicable to their local context and responds to the needs of their constituencies.1 Evidence synthesis methods support the adoption and translation of evidence by using rigorous methods to identify, select and analyze relevant evidence. The most well-known among these methods is the systematic review, a scientific investigation that focuses on a specific question and uses explicit, preplanned scientific protocols to identify, select, assess, and summarize similar but separate studies.2

Systematic reviews often address very specific questions and focus on a narrow scope of populations and interventions, which may enhance the validity of the report’s findings but can also reduce its applicability and uptake (Israel 1998). Another barrier to uptake of evidence by decision makers is that systematic reviews are often reported as long, text-heavy documents. Many decision makers have been shown to be more receptive to evidence presented and summarized graphically with emphasis placed on the most important (or actionable) results and minimization of methods details.3,4

This case study describes a collaborative study started in 2016, when the Rhode Island Arts and Health Advisory Group (hereafter: “The Advisory Group”) convened to develop a set of policy, clinical practice, and research recommendations to be implemented by the state Department of Health and to promote the role of arts in health more broadly. Arts-based interventions can include many different artistic modalities, such as music, dance, theater, literature, visual art, architecture, interior design, and are designed to enhance health care, promote public health and support community well-being.5 These interventions are delivered by creative arts therapists and include expressive arts therapies and artists in health care settings and are accessible across the continuum of care, from in-patient to community-based programs.5 An emerging evidence base demonstrates the efficacy and comparative effectiveness of these interventions relative to more traditional health interventions (eg, drugs), as well as the positive role that arts-based interventions can play in affecting social determinants of health such as social isolation, resilience, and personal self-efficacy.

The Advisory Group—comprised of artists, clinicians, community members, and patients—decided to develop an evidence map of arts-based health care interventions to better understand the quantity and quality of existing evidence, to establish gaps in evidence and identify areas where sufficient evidence exists to develop systematic reviews or actionable evidence-based policy. Evidence mapping is an emerging research synthesis technique that employs a systematic and replicable methodology to aggregate and organize data about heterogenous or complex interventions in a logical way, primarily as a technique for identifying latent connections or patterns of effect between interventions and populations that can be used to inform policy—and systems-level questions. The evidence map method is especially useful for categorizing the state of the literature in order to generate hypotheses—that is, motivating the need for further reviews and/or primary research by identifying gaps in the evidence. In contrast to traditional quantitative methods of evidence synthesis, such as statistical meta-analyses, evidence mapping relies on the tabular, qualitative and descriptive synthesis and is often designed to characterize the quantity and quality of a body of literature without formal quality or risk of bias assessments using study design features.6 As an analytic technique, evidence mapping thus prioritizes breadth and scope over precision. But this trade-off makes the approach potentially far more useful for decision makers whose purview is broad or concerns a heterogenous population, such as health system administrators, public policy makers, and research funders.

During initial meetings between researchers and the Advisory Group, the team deliberated the benefits and limitations to evidence synthesis in the policy context and determined that 2 modifications should be made to improve applicability, uptake, and dissemination of evidence synthesis in our project. First, we adopted a community-engaged approach to evidence synthesis and provided training and mentoring to our community research partners (CRPs) to codesign, coproduce and codisseminate an evidence map. Second, we recognized that to disseminate the findings of our research, the report and results of our research would need to be accessible beyond the walls of academia. We agreed that publication in a journal was desirable, but the first obligation was to identify a means of dissemination that would inform policy makers, health care providers and remain accessible to our CRPs. The group decided to develop an interactive, online evidence map in collaboration with the Aero Data Lab, a data science, ethics, and policy consulting group affiliated with the Harvard Medical School (www.rhodeislandartsandhealth.com/research-database).

In what follows, we outline and elaborate our approach for this project, innovating on well-established methods of evidence synthesis by: (1) training and mentoring CRPs; and (2) developing an online, interactive evidence map to support dissemination and implementation of evidence.

METHODS

Our community-engaged approach to evidence synthesis included both stakeholders and CRPs participating in the development and implementation of evidence synthesis protocols. Stakeholders are routinely engaged in systematic reviews to provide topical expertise, improve the identification of pertinent evidence and emerging outcomes of import, informing and consulting at regular intervals. Our project relied on a complement of stakeholders that represented local health care organizations, health policy and governmental offices, not-for-profit organizations, arts and humanities organizations working with health care organizations and providers, and local artists. Selection of stakeholders was guided by the 7P framework devised by Concannon et al.7 The role of each stakeholder was to inform our work, meeting with us at 2 different time points: first, to review and comment on the protocol and interim findings, and second, to review our recommendations and interactive evidence map.

The CRPs were invited to participate as research partners on the project, collaborating on both the study design and research process itself. The community-engaged approach draws upon a number of research frameworks, including community-based participatory research, participatory action research, and patient-centered outcomes research. These approaches share several common themes: (1) engaging community members in choosing research topics, developing projects, collecting data, and interpreting results; (2) acknowledging the importance of existing systems (social, political, cultural, and economic) to health behaviors and outcomes; (3) emphasizing both qualitative and quantitative research methods; and (4) prioritizing translation of the findings into practice and policy.8

Training Community Partners For Successful Community-engaged Evidence Synthesis Research

The training team consisted of 2 research methodologists with expertise in evidence synthesis methods (S.S., V.R.), an Associate Professor of Emergency Medicine at the Alpert Medical School, Brown University (J.B.), the Chief Administrative Officer of the Rhode Island Department of Public Health (S.B.), the Director of Arts and Health, Rhode Island State Council on the Arts (S.B.). The training program adapted expert guidance from the Institute of Medicine, Agency for Healthcare and Research Quality Effective Health Care Program, and the Cochrane Handbook for Systematic Reviews of Interventions.9–11 for the community audience.

Over the course of the 18-month project, members of the group learned evidence synthesis methods (eg, developing key research questions, search strategy, screening abstracts, data extraction) through didactic training and mentorship via the Brown University School of Public Health Evidence-based Practice Center. The training included 3 phases: (1) Considering Evidence Synthesis-based Approaches; (2) Research Protocol Development; and (3) Evidence Synthesis and Policy Recommendations. The research team (CRPs and training team) convened monthly meetings for 1.5–2.5 hours. Each session included 30–45 minutes of lecture/discussion designed to introduce a key evidence synthesis concept, method or technique as described in the learning objectives (Fig. 1). Remaining time in each session was devoted to interactive activities to inform decision-making around protocol details and strategic planning for the use and implementation of the evidence. Initial sessions provided opportunities to discuss the concepts of evidence-based medicine, what constitutes evidence in health care, how quality is considered in the context of peer-reviewed literature and the role of gray literature in evidence synthesis.

FIGURE 1.

Training was completed in 3 phases (left, gray boxes) covering important topics (right) delivered over the course of 12 sessions.

The second phase focused on the codevelopment of the evidence map protocol. This phase introduced CRPs to the 6 steps of the evidence synthesis process with relevant examples, detailing how to make protocol decisions related to preparing the research question, developing the search strategy, screening relevant abstracts, extract data, analyzing the evidence and reporting findings (Fig. 1). The team was responsible for learning the steps, making decisions for conducting the research and subsequently completing each task. Team meetings were focused on supporting each of these 3 objectives. The team was committed to making decisions through consensus, normative group technique was employed to introduce the topic, frame the question, provide time for introspection, promote discussion, and come to a consensus. To support the conduct of the protocol outside of the research meetings, the methods leads (S.S. and V.R.) created recordings using a webinar platform to demonstrate each task (eg, searching literature, screening abstracts) and circulated these recordings to the team so participants could review at their own pace.

Development of on Online, Interactive Evidence Map Guided by Community Input

The initial product of our evidence review was a spreadsheet of data elements extracted from the published literature. We established a partnership with Aero Data Lab (a project founded by SPH), which is an international collaboration of scientists, ethicists, and policy makers developing interactive, online evidence synthesis and data visualization tools intended to enhance the transparency and efficiency of the research enterprise. One of the visualization methods developed at Aero Data Lab is an “outcome matrix,” which plots a sample of studies or trials by primary study outcome along with the x-axis and intervention type along the y-axis. We adapted this method, using arts-based interventions categories for the x-axis and patient populations for the y-axis, to create an online, living evidence map of arts-based interventions in health care (Fig. 2). The current evidence map can be found at www.rhodeislandartsandhealth.com/research-database.

FIGURE 2.

The figure is a screenshot capture of the outcome matrix available as part of the interactive, online evidence map. The y-axis represents the populations and x-axis represents the types of arts-based interventions, coded with support of our community research partners and stakeholders from information provided by study authors. Each bubble represents an individual study identified in the literature, the size of the bubble indicates the total number of participants in each study. Bubbles are semitransparent and darker colors indicate multiple studies testing the same intervention/population combination. ALS indicates amyotrophic lateral sclerosis; COPD, chronic obstructive pulmonary disease.

Refining the evidence map required discussion and iteration led by the CRPs. Training in evidence synthesis and screening the abstracts prepared the CRPs for the first discussion on how to code the data, to distill the messages of the evidence into something accessible to the reader but not so much that the message is lost. The CRPs were very keen to “embrace the messiness” as they would say, that this field includes many nuanced approaches to integrating arts into health. They admonished that there is a danger in oversimplification for the sake of neat packaging. A number of CRPs made note that the training they received made them feel more comfortable in these conversations and “pushing back” on the researchers at the table. For example, the CRPs expressed concern that the “interventions” did not fully describe the use and uptake of arts-based interventions in health care. The first version of the map categorized the interventions (x-axis) and populations (y-axis) according to detailed descriptions in the article abstracts and full text. However, this produced an unwieldy taxonomy, often conflating arts interventions delivered by trained art therapists with those programs designed and delivered without an art therapist. This obscured important insights about patterns of effect and outcomes within and across categories.

We thus worked collaboratively to identify a taxonomy of interventions that was robust enough to capture sufficient detail and resonate with the local practice, but not so detailed and heterogeneous that it rendered the map incomprehensible. In this way, the CRPs provided critical feedback on the proposed protocol for data extraction and worked collaboratively with researchers to build a taxonomy that reflected both the available evidence and the nuance of how these interventions are applied in the community. The community level perceptions are important to building an evidence base poised for translation and dissemination. Absent a robust understanding of how evidence synthesis worked, the CRPs may not have raised the issue, deferring to the research team, or would not have been able to effectively collaborate to find an appropriate solution to their concerns.

RESULTS

Our CRPs screened over 6000 studies to ultimately develop an evidence map yielding 481 studies, including ∼30,000 participants across all age ranges. The most frequently studied populations include behavioral health, cancer patients, and aging populations, which align with priorities set forth by the Rhode Island Department of Health. The synthesis of the evidence will be published in a forthcoming manuscript with our community partners as coauthors.

The process of developing the protocol and reviewing the literature with the CRPs led to robust discussions, driven by evidence and supported by the local context. For example, the map yields >80 studies on the role of dance in neurodegenerative diseases, such as Parkinson disease. One of our CRPs provides community-based dance classes for Parkinson patients. Understanding the strengths and limitations of evidence has led her to collaborate with local neurologists to consider whether sufficient evidence exists to develop local clinical practice guidelines for the use of dance in this population and if these studies could be used to support coverage of dance classes for these patients. More generally, the CRPs were able to review the evidence and identify that many arts interventions and populations participating in arts programming were not currently represented in the literature. This is a valuable insight for CRPs and stakeholders alike, as it clearly establishes where there is a need for further research.

The interactive evidence map serves as a tool to navigate the relevant, existing literature and the key features were informed by our CRPs. The map provides information on studies, locating them by the population of interest on the y-axis, coded by their condition (ie, “Cancer patients”), by stage of life/age (ie, “Children”), or by their role in health care (ie, “Caregivers,” including clinical and family caregivers) derived from the text of the publication. This coding was developed by the CRPs to support the use of the map by health care providers and public health professionals in the state as they consider interventions for high priority populations, but also patients and families interested in what works for them–however, they define themselves. The x-axis represents the types of arts-based interventions, coded with support of our CRPs and stakeholders from information provided by study authors.

Each bubble at the nexus of the population/intervention descriptors represents an individual study identified in the literature, the size of the bubble indicates the total number of participants in each study. Bubbles are semitransparent and darker colors indicate multiple studies testing the same intervention/population combination.

Using the cursor, the user can hover over a bubble to reveal further detail about the studies. These details were considered as useful to all potential end users without presenting too much information that may be useful to only a portion of the audience. We take for example studies on Parkinson’s Disease and Dance. Study details include the PubMed ID (if available), Condition Detail (if provided in text), Number Enrolled in the Study and Intervention Detail, if provided (in this case, the paper is about a specific type of dance—Tango—and is therefore specified). Clicking on a particular study will automatically link to the abstract in PubMed.

The evidence map is accessible and available online (www.rhodeislandartsandhealth.com/research-database) and, as part of the next phase of the project, will remain updated with new studies and the CRPs are working to add new functionality to make the map more meaningful to end users.

The evidence map also helped to demonstrate the importance of high-quality study design and reporting. For example, our CRPs observed that many studies could not be included on the map because they failed to adequately describe their populations, interventions, or results. Other studies were included on the map, but only reported outcomes that were limited import to patients and communities or did not seem to represent the lived experience or patients. Our CRPs noted that this opportunity to critically appraise the medical literature gave them a great perspective on how studies can be designed to resonate more fully with the needs of patients and communities. The CRPs were surprised at the number of studies that only captured clinical or biometric outcome measurements, for example, studies of dance for patients with neurodegenerative disease. This was seen as a missed opportunity to capture psychosocial outcomes, dimensions of health care utilization and measures of caregiver well-being.

Where many studies existed, the conversation included discussion of quality—specifically how to measure quality in these studies, how study quality would impact their decision-making. These robust conversations were aided by having the interactive tool to provide examples of key evidence synthesis.

The inclusion of CRPs requires an extra commitment by research methodologists to ensure they are trained and supported throughout the process. In our project, attrition was low, and our group maintained a quorum throughout. The commitment required in terms of time and effort are not nominal investments by CRPs or researchers. Translating evidence into policy and practice is often a challenging endeavor and it is often easy to forget that communicating and translating the methods and techniques of evidence synthesis—identifying, selecting, assessing and analyzing evidence—is important as well.

For the government stakeholders involved in this work, the evidence map was used to support the development of public health and clinical practice recommendations and to inform their implementation strategies. These include the alignment of the arts and health evidence map with the state population health goals to determine if existing evidence can be leveraged to support current public health programming, promoting evidence-based arts-based interventions as part of clinical practice and developing collaborations between artists and researchers to document the role of arts in health care and well-being.

Finally, the interactive map provided the CRPs with user-friendly access to the results of their work. Advisory Group members have already utilized the evidence map to support grant applications to promote the uptake of arts-based interventions to address the opioid crisis and have received funding to fill evidence gaps identified by the map.

Our project evaluation included an online survey and debriefing sessions after each phase of the process and as indicated by the CRPs. The survey measured indicators of engagement in research as described in the Patient-Centered Outcomes Research Institute (PCORI) Engagement Strategy Rubric (ie, reciprocal relationships, partnership, colearning, transparency, honesty, and trust).12 The survey revealed that our approach to evidence synthesis measured high on all aspects of engagement. Participation also improved CRPs confidence in engaging with the health care system, developed greater empathy and understanding of others in the community and increased interest in using science or research in advocacy efforts. We also measured the tangible skills developed as part of our training, with most participants indicating that they feel confident in their abilities to develop a research question, search the medical literature, read a journal article and identify a population, intervention or outcome in a journal article. The strongest factors that motivated participation in this work were helping patients receive better care, belief that participation can result in meaningful findings, receiving information on the research and how it is conducted. The majority of our CRPs indicated that they are more likely to collaborate with researchers after this experience.

CONCLUSIONS

Our case study demonstrates the feasibility and utility of training CRPs in evidence synthesis methods to promote the uptake and dissemination of evidence within the context of public health policy and clinical practice. Stakeholders in evidence synthesis play an important role in consulting and informing on the composition and direction of the protocol and final reporting. The unique role of the CRP complements the stakeholder role by empowering nonresearchers to participate in the research process. Our work indicates that community-engaged evidence synthesis promotes community dialogue and engagement in research, specifically towards: (1) elucidating outcomes of import to patients and communities that are not represented in the medical literature; and (2) identifying comparisons among interventions that resonate with patients and communities.

Translating the findings of evidence synthesis products into an interactive, online tool enabled richer engagement with the parameters of the map. The online map served an important role in fulfilling the commitments made by the researchers to ensure that products of collaboration were accessible and meaningful to community partners. The stakeholders and audience of our report have demonstrated that an accessible evidence map allows them to explore the literature and understand the breadth, distribution, and gaps in evidence.

There is an intrinsic and extrinsic benefit for researchers mentoring others in evidence synthesis and for community partners acquiring the set of skills necessary to critically interpret evidence and complete an evidence report. Or as one of our CRPs put it: “This project has allowed me to put something back into the system that has nourished me, personally and professionally, for decades. Since entering this world of scientific research in our field, I view my interactions with clients from a more rigorous and [with a] long-term perspective.” —Community Research Partner, Rhode Island Arts & Health Advisory Group.