Training Health Policy Researchers on Policy Engagement and Research Translation for Greater Impact: Evaluation of the Amplify@LDI Program

Abstract

Background

Few researchers receive formal training in research translation and dissemination or policy engagement. We created Amplify@LDI, a training program for health services and health policy researchers, to equip them with skills to increase the visibility of their research through translation and dissemination activities.

Aims

To describe the program’s participants and curriculum, and evaluate the first 2 years of the program.

Setting

The Leonard Davis Institute (LDI) at the University of Pennsylvania (Penn).

Participants

An annual cohort of 12 LDI Senior Fellows (Penn faculty) from multiple schools, disciplines, and ranks at Penn.

Program Description

The Amplify@LDI curriculum includes 6 sessions on different aspects of research translation and dissemination, including media and social engagement, writing Op-Eds, and policy engagement.

Program Evaluation

Participants reported measurable increases in time spent on translation and dissemination activities, as well as new enthusiasm for and confidence in policy engagement. Participants’ reach (as measured by Altmetric) increased during the program, compared to smaller increases or reductions in reach for two comparator groups.

Discussion

In our preliminary evaluation of Amplify@LDI, we find strong evidence of positive impact from participant evaluations, and suggestive evidence that participation in the program is associated with significant increases in the reach of their research.

Supplementary Information

The online version contains supplementary material available at 10.1007/s11606-023-08341-x.

Introduction

In academic medicine, it is more important than ever for researchers to get their findings into the public domain and into the hands of people who can use them. It is no longer sufficient to just “do the science” and hope that research insights reach intended policy and practice audiences. Health researchers are increasingly asked to translate research into plain language and disseminate key findings to policy makers and practitioners for maximum impact¹. However, few researchers receive formal training in research translation and dissemination or policy engagement. Many (particularly junior faculty pursuing tenure at research-intensive universities) lack incentives and confidence to undertake what can feel like high-risk, low-reward “extra work” ^2,3.

To address this gap in translation expertise, we created a training program for health services and health policy researchers to equip them with skills to increase the visibility of their research through translation and dissemination activities. Launched in Fall 2020, the annual program (Amplify@LDI) is hosted by the Leonard Davis Institute of Health Economics (LDI), a hub for health policy research at the University of Pennsylvania. LDI’s mission is to catalyze and facilitate research to improve the nation’s health care delivery, health policy, and population health. Faculty at the University of Pennsylvania can apply to be affiliated with LDI (i.e., become an LDI Senior Fellow) for 3-year renewable terms if they are actively engaged in health policy research. Consistent with LDI’s mission, Amplify@LDI aims to elevate LDI Senior Fellows as content and policy experts by amplifying the reach and impact of their policy-relevant research. The program builds capacity in and enthusiasm for the critical step of translating and disseminating research to broad national audiences. Amplify@LDI is a cohort-based program of 8 two-hour virtual and in-person sessions held over an academic year. The program aims to build skills in translation and dissemination of research, including engaging with the media and policy makers, developing a social media strategy, and writing for lay audiences. Below we describe the program’s participants and curriculum, and evaluate the first 2 years of the cohort program (2020–2021 and 2021–2022) using quantitative impact and qualitative program evaluation feedback.

Setting and Participants

Amplify@LDI is a signature program of the Leonard Davis Institute (LDI) at the University of Pennsylvania (Penn). LDI supports more than 400 faculty members (“Senior Fellows”) across Penn’s 12 schools and affiliated health systems with a range of programming and services, including research infrastructure, pilot grant funding, convenings including seminars and national conferences, training in grant writing and research methods, and networking. In this context, LDI recruits a new cohort of Amplify@LDI participants each year. A Call for Applications is circulated to all Senior Fellows in late spring. The brief application asks Fellows to respond to three prompts:¹ In 3–4 sentences, please describe your program of research (100 words or less);² What motivates you to apply to the Amplify@LDI Program? What do you hope to get out of the program (250 words or less)?³ Please share a positive or negative experience you've had with translation, dissemination and/or policy engagement activities (250 words or less). A selection committee comprising LDI staff and faculty directors reviews and scores applications individually and then meets to make the final selections. Key selection criteria include demonstrated interest in research translation and dissemination and a policy-relevant program of research. We also seek to form a diverse cohort of 12 LDI Senior Fellows with respect to academic field, school, academic rank, methodological approach, and prior experience with research translation and policy engagement.

Over the past three recruitment cycles, we have averaged 18–19 applications per cycle. The 2020–2021 Amplify@LDI cohort included participants from Penn’s schools of Medicine (7 participants), Nursing (2 participants), Engineering and Applied Sciences (1 participant), Arts and Sciences (1 participant), and Law (1 participant). The 2021–2022 cohort included participants from the schools of Medicine (8 participants), Nursing (2 participants), Arts and Sciences (1 participant), and Communications (1 participant). Across the two cohorts, 13 were assistant professors, 4 associate professors, and 7 professors; and 16 female and 8 male participants. Representation of Medicine and Nursing faculty in the program are proportionate to their representation in the larger Penn LDI Senior Fellow population.

Program Description

The Amplify@LDI curriculum is designed (and revised) to meet the program goals of building confidence and concrete skills in research translation and dissemination and policy engagement. Each of the program’s content sessions focuses on a different aspect of research translation and dissemination, including Media Engagement and Interviewing; Increasing Social Media Reach; Writing and Placing Op-eds; Writing Blogs and Briefs; and Policy Engagement. The program includes hands-on training, practice assignments, and editorial support. Sessions are led by topic experts at Penn (e.g., for policy engagement and writing briefs and blog posts) or outside trainers (e.g., for social media outreach). Sessions are a mix of didactic presentations, discussion, and hands-on practice. Prior to each session, participants are asked to complete brief preparatory assignments in accordance with the topic (e.g., to draft an Op-Ed). Some sessions include an opportunity for a brief one-on-one meeting with the presenter. In 2020–2021, the program was conducted virtually. In 2021–2022, sessions were hybrid, with participants having the option to join either in person at LDI’s central location on campus, or virtually. Most hybrid sessions were about evenly split between in-person and virtual participation.

Program Evaluation

The Amplify@LDI program is rigorously evaluated and revised annually in response to participant and faculty and staff feedback. Following each session, participants complete a brief survey that includes questions about most enjoyable aspects of the session, key learnings, suggested changes, and concrete action steps for implementating learnings. At the end of the annual program, participants complete a comprehensive evaluation that captures actions taken and progress made on each of the session topics, the most useful aspects of the program, suggestions for improvements on any aspect of the program, and ideas for new topics (see Supplement S1 for evaluation survey). Data from both surveys are collected, analyzed, and reviewed by LDI staff after the program completion each year in order to revise and improve the program in future iterations. Evaluation of individual sessions shows that participants benefitted from the didactic, discussion, and hand-on practice portions of Amplify@LDI sessions. They also appreciated the opportunity to learn from and build relationships with the experts who led the sessions. Examples of actions taken following the training sessions include writing, submitting, and publishing OpEds; doing media and podcast interviews; and building a stronger social media presence. In response to the policy engagement sessions, participants report attending lobbying days and direct advocacy with legislative staff, applying to serve on committees (i.e., Medicaid and CHIP Payment and Access Commission), and connecting with relevant advocacy groups and regulatory agencies.

Evaluations of the overall program were also very positive (see quotes from the year-end evaluations of the 2020–2021 and 2021–2022 cohort in Supplement S2). Participants reported having learned about the wide range of opportunities for engagement, translation, and dissemination beyond journals and conferences as well as about the concrete steps and skills and available strategies and tools to support them. Amplify@LDI helped participants to evaluate which dissemination activities to pursue depending on the content, intended audience, and their own capacity and interest. Participants reported measurable increases in time spent on translation and dissemination activities, as well as new enthusiasm for and confidence in adding these activities to their busy research careers. Participants also reported eagerness and willingness to incorporate reach and engagement metrics in tenure and promotion statements and summaries.

The Amplify@LDI program has been revised based on participant, staff, and faculty feedback. For example, the presenters and topical emphasis of the policy engagement session(s) are updated to best meet participant needs and interests. A storytelling skills session was also added, reflecting the importance of narrative in communication scientific results. A new session “Choose Your Own Adventure” asked participants to select a translation, dissemination, or policy engagement skill or activity not covered in the Amplify@LDI program and taught that topic to their peers. Participant-selected topics have included developing whiteboard animations to share study results, appearing on podcasts, creating videos and reels for social media platforms, and hosting a dissemination workship in a global setting.

To quantify the effect of Amplify@LDI on the reach and impact of participants’ research, we also undertook an analysis of the change in the reach of participants’ (n = 22) research before and after the program, compared to a similar group of faculty who are also Senior Fellows but did not participate in the program (n = 22). We matched each Amplify@LDI participant in 2020–2021 (n = 11) and 2021–2022 (n = 11) to another Penn faculty member who did not participate, matching where feasible on academic department/school, rank, and gender, creating Comparison Group #1 (CG1). To test the sensitivity of the results to comparator selection and to account for regression to the mean, we created a second comparison group (CG2), matching on baseline level of reach, as measured by the total number of mentions (described below).

We measured reach using Altmetric, which tracks online activity related to scholarly research output and generates what Altmetric calls “attention” metrics: exposure to, engagement with, and dissemination, influence, and impact of scholarly research⁴. For each Amplify@LDI participant and matched comparators, we used Altmetric Explorer⁵, which uses a proprietary algorithm to quantify and report the number of research outputs (described by Altmetric as journals articles or published datasets) and the total number of mentions (described by Altmetric as an online article, document, or social media post that references a research output)⁶. To standardize mentions across researchers with varying productivity, we then calculated the total number of mentions/research output for the academic year prior to and during Amplify@LDI participation. Our main outcome is the difference (between Amplify@LDI participants and comparators) in mean change in mentions and mentions/research output. We tested for statistical significance in this difference between groups using a Wilcoxon matched-pairs signed-rank test.

Our analysis shows that Amplify@LDI participants had lower baseline mentions and mentions/research output compared to CG1 but more similar baseline metrics to CG2, which was matched on baseline reach (Table 1). During their Amplify@LDI year, participants’ mentions increased 177%, compared to a 27% decrease for CG1 (p < 0.001) and a 32% increase for CG2 (p = 0.07) (Fig. 1A). Participants’ mentions/research output increased by 33%, vs. a 57% decrease for CG1 (p = 0.02) and a 33% decrease for CG2 (p = 0.07) (Fig. 1B).

Table 1.

Characteristics of Study Cohort and Comparison Groups

	Amplify@LDI program participants	Non-participants
		Comparison group #1 (CG1)*	Comparison group #2 (CG2)**
N	22	22	22
Female (%)	68	64	59
Academic rank in baseline year (%)
Assistant Professor	59	64	56
Associate Professor	14	5	14
Professor	27	32	32
Altmetric metrics in baseline year, mean
Total mentions	314	722	313
Total mentions/research output	21	67	43
Altmetric metrics in Amplify@LDI year, mean
Total mentions	870	524	412
Total mentions/research output	28	29	29

^*Cohort matched on gender, rank, and department/school

^**Cohort matched on baseline total mentions

Figure 1.

Total mentions (A) and mentions per output (B) before and after Amplify@LDI participation for participants of Amplify@LDI and two comparison groups.

Discussion

Research translation and dissemination and policy engagement are no longer optional competencies or activities for health services researchers. Department chairs, tenure and promotion committees, and funders routinely look for evidence of reach and impact of research findings beyond peer-reviewed publications. In our preliminary evaluation of Amplify@LDI, an innovative cohort-based research translation and dissemination program at the University of Pennsylvania’s Leonard Davis Institute of Health Economics, we find strong evidence of positive impact from participants evaluations, and suggestive evidence that participation in the training program is associated with a significant increase in the reach of their research (as measured by Altmetric Scores) relative to matched comparators.

Our program evaluation is limited by the self-report nature of the feedback from participants on the evaluation survey, which may be subject to social desirability bias, reciprocity bias, or optimism bias. A possible limitation to our quantative comparison of reach metrics is that we cannot control for participant selection into the program; selecting participants with low research impact or less interest in research translation would make it easier to detect relative increases in reach compared to peers. However, our actual selection process is more likely to bias results to the null, given that we screen applicants for policy-relevant research agendas and demonstrated interest in media and policy engagement.

We will continue to offer the Amplify@LDI program to successive cohorts of LDI Senior Fellows going forward. We recognize that the curriculum will need to evolve to meet changing needs of participants and new realities of translation and dissemination technologies and processes and there are opportunities to scale the program to reach a larger group of researchers. Inclusiveness, cost-effectiveness, and sustainability of the program are also critical strategic issues to consider going forward. As a leading academic institute supporting cutting-edge health services research, we believe that focused capacity-building in research translation and dissemination is a core offering that can effectively and efficiently increase the reach and impact of our scholars’ research. We hope that Amplify@LDI can contribute to broader efforts in the health services research community to build critical capacity in research translation and policy engagement.

Supplementary Information

Below is the link to the electronic supplementary material.

Data Availability

De-identified program data may be made available upon request to the first author.

Declarations

Conflict of Interest

RMW reports funding from the Leonard Davis Institue of Health Economics and from the National Institutes on Aging. None of the other authors report any conflicts of interest.