Table 2.

Participant quotes

Theme	Illustrative Quote(s)
Benefits of using participatory systems science methods in implementation science to researchers
Improved relevance of research	“…like a lot of participatory or different types of community engaged stuff, it helps you like, know what not to waste your time on. Like, you might think something’s this really great idea or this thing that it’s like, well, the literature says, these are the best interventions and let it out. And people are like, No. And so it kind of does can help you hone in on that and what they need.” (Participant U) “…getting to the bottom of the problem, is actually the biggest problem is the need for the simulation model that we have. And we only learned that through engaging people who are misusing opioids. So that I would say the models are becoming a lot more realistic.” (Participant L) “You can be talking with them and say, ‘how would you use this?’ You know, ‘would you actually ever present any of this at a city council meeting?’ Or ‘would you actually ever put this if you’re talking, let’s say to CDC, would you actually ever put this in your RFAs for the states?’ And one, you’re getting them to think about that, but also, you’re getting them to tell you, Oh, they actually might put this in the RFA. So let’s make sure that what we’re building is going to be relevant for that.” (Participant M)
Build and strengthen relationships	“So it [engaging partners in modeling] definitely spurred I think, bigger discussions that don’t ever happen, because it’s not in the data.” (Participant D)
Expand into areas with limited data	“…because … looking at things where you have a limited amount of data, you know, there’s going to be a variation about all the different parameters that you’re not going to have a point estimate of anything. But you’re going to have some idea of how often it happens.” (Participant C)
Structure and name for engagement activities	“… the NIH will fund you to do it, right. I mean, like, you get publication out of it, you get grants out of it, instead of just being like, we work with community partners, we did consensus building, … but if you start putting these, like sort of more technical terms on, but it’s really the same type of process, you know, it’s just so that’s a little bit like, it’s a little bit like pessimistic, but it’s true.” (Participant T)
Benefits of using participatory systems science methods in implementation science to practitioners
Improved ability to think in systems	“Yeah, no, we did some things [before partnering with researchers to use systems science]. But it was too this is what we need the overall picture, because this can lead us on for years. We can work on this for a long time… running around like, a chicken with their head cut off…, but this gives us a goal to work to an overall, you know, picture, and this is what we were missing.” (Participant O) “…a difference would be that it helps people really intentionally think about the consequences of what they’re proposing. And this could be good or bad consequences.” (Participant D) “they’re considering their own their own things that they want to accomplish based on others in their community, so that they’re not duplicating effort, they’re hopefully addressing gaps that already exist. And if there are others doing work, rather than duplicating they can work together to hopefully have a better chance of having an impact, because they’re joining together to do that.” (Participant I)
Gain resources for subsequent advocacy and financial support	“…some of these knowledge translation documents that we’ve built… based on some of these dashboards were used in ***** in the city council meeting to help pass stronger tobacco control legislation.” (Participant M) “People actually ended up copying and pasting some of the visualizations into their grant applications to funders, to potential funders and into reports to existing funders. And that was that was interesting. I hadn’t anticipated that people would use it for that. But people did.” (Participant G)
Common benefits to researchers and practitioners
Visual, intuitive nature of systems science methods	“…one nice thing about agent-based modeling and like thinking in those terms is it does, it’s naturally very intuitive to a practitioner, because they are dealing with someone’s immediate decisions and behavior. And so our partners are really excited about that part of it.” (Participant H) “We didn’t know how to put it in on paper or in graphical format, or even an order. So the benefit of having that [causal loop diagram], and being able to look at it, and follow it is pretty amazing. Because it’s, it’s not so jumbled up in your head, you know?” (Participant O) “I think that agent based modeling offers a lot of advantages for engaging with stakeholders, because it can be very visual, because it’s role based, because you can usually explain what’s going on in and even in quite a sophisticated agent based modeling without showing any math, no P value tables, no equation and you can tell stories about where you personified individuals who you’re following through the model and then all of my work I take full advantage of that capacity and agent based modeling to engage with stakeholders.” (Participant K)
Ability to integrate lived experiences directly into research products	“… there are opportunities, especially if you’re careful about how you structure like explaining what the models are, what you mean by inputs, … where you really show them, this is where like your direct input, like directly informs what emerges. That’s… a nice touch.” (Participant N) “…And then later, when they were looking at the map [can] be like, ‘Oh, I literally told you that story. And like, it’s right here.’ And that happened, like multiple times. And I mean, they maybe didn’t realize that they weren’t the only people who told us that story. But they were like, oh, that’s what I said. And we’re like, yeah, we listened to you. So that was powerful.” (Participant F)
Facilitators to using participatory systems science methods in implementation science
Established, deep relationships with partners	“…because of that trust, and that long standing relationship, … we’re very transparent, and we try as much as we can to distribute funds as equitably as we can, as each different type of grant mechanism will allow. So like, I think we’ve gotten to a point where the community partners trust us, they know we live the grants life, they know that a lot of the support we provide to them and what we’re able to do is only like, driven by, like our ability to successfully get grants that then drive the work. And so I don’t think there needed to be like, like, let’s sell you on the system science piece, they trust us methodologically” (Participant N)
Actively make systems science projects accessible	“But the idea is, is to visually kind of get across lots and lots of complex things that are happening in hard math behind the scenes in a really user friendly, intuitive way. So the more successful you are at that the more capable you are communicating, hey, we’ve built up this model that captures all these relevant aspects of the way that the world actually works, [and it] lends credence when you when you finally show your results again… You know, people are gonna want to know what’s going on behind your model, going on behind the scenes in your model. And if they’re, they’re kind of shown just the hard numbers and the equations, I found that a lot of people kind of, they’ll back away, they’ll kind of feel kind of bullied away, almost.” (Participant J) “By having that visual, they can immediately sort of jump into and understand what you’re thinking about, and immediately start to contribute. So I, I think, maybe the diagram like, necessarily gets at, like, all of the complexities involved… And I think people are, like, often, like, even surprised, like pleasantly surprised by their own ability to like, really jump into that fast, … and that’s both like patients and providers and other types of stakeholders.” (Participant P)
Teams with diverse training and skills	“Think about it as like a research skill and have somebody on their team that has that skill. Because it’s kind of like qualitative work. Everyone thinks that you can just do it. Yeah, I mean, you can physically, [but] it’s not gonna be a good idea, which [emphasizes] that there is a knowledge base and there is a skill set. And so just to have, if you’re not that person, have somebody on your team who knows this world, because it’s not just like a tack on.” (Participant Q)
Practice-driven research focus	“From a research perspective, like maybe generalizability people worry about, like whether or not there’s an influence on, you know, is this truly objective research and that sort of thing. But then we also would possibly argue that we’re getting more people to participate in the work of informing, because we’re removing some of those barriers.” (Participant E)
Funding to support engagement and compensate participants	“We set up a table and we like make sure that people get something, like are well compensated, but not to a point where it feels coercive that they, you know, sit with us… Well, I think the things we definitely need to work on are the how to roll it [project outputs] back out to them, how to share back with them how to not just extract but also you know, change something for them.” (Participant E)
Future directions to improve the use of participatory systems science in implementation science
Strengthen the foundation and reach of systems science	“I think it would be helpful to have some… thoughtful people thinking about like, where does the engaged systems pieces fit into… the existing frameworks that are like, thinking about, like implementation… [or] public health practice. I kind of see those overlapping often… So there’s like the frameworks for like, how you do implementation, how you do in your basic public health practice, strategic planning kind of work, like I think, some thought into, right, engagement with these systems pieces. Where does that fit into those frameworks?” (Participant A)
Recognize and reward team science	“The main thing that would help the most is interdisciplinary teams or team science. At least in my experience, I rarely see one individual who is capable of truly engaging with communities and good at-and can do research and has the technical expertise to do systems science modeling… three really different skill sets, and it helps tremendously if you can have people who can serve in and are really robust in those fields in and of themselves and not be expected to do all of the things” (Participant H)
Training and professional development	“I think the problem right now is that, like, I was trying to get into participatory group model building and trying to find someone who could actually do that. There’s like, less than 20 people around the country who actually do it. Right. So there’s just a lack. There’s no, there’s no pipeline… Now that there’s a lot of people that do CBPR [community-based participatory research], that’s the value that could potentially, you know, see the value in system science approaches. And then there’s like a ton of like, engineer trained system science modelers that are incredibly skilled, but like, the people that are in that overlapping Venn diagram are like, there’s like five people in the country. You know, and so there’s, there’s like a pipeline problem.” (Participant T)
Funding that suits the nature of participatory systems science work	“And that’s hard to get through NIH, because reviewers want to know that you have all the data that you need. And that, you know, you have that you know how to do this project. And I don’t want to structure the model fully until I have had all the conversations, you know, in the community to be able to do it. So NIH is hard for this type of work” (Participant B)
Protected time for engaging community partners	“You need this time in this space to really build those relationships, build a track record, think about how you want to, you know, facilitate all of this, be able to show people what it looks like” (Participant B)
Improving dissemination and accessibility of products	“…making the outputs more accessible. And sharing findings…[through] really cool, user friendly, interactive tools that you can put up on the web, like things like that. There is probably a shortage of people who can create those things.” (Participant W)