The science of stakeholder engagement in research: classification, implementation, and evaluation

Abstract

In this commentary, we discuss the science of stakeholder engagement in research. We propose a classification system with definitions to determine where projects lie on the stakeholder engagement continuum. We discuss the key elements of implementation and evaluation of stakeholder engagement in research posing key questions to consider when doing this work. We commend and critique the work of Hamilton et al. in their multilevel stakeholder engagement in a VA implementation trial of evidence-based quality improvement in women’s health primary care. We also discuss the need for more work in this area to enhance the science of stakeholder engagement in research.

With the uptake of implementation and translational sciences, the scientific community is shifting the focus from pure scientific discovery to include the translation and implementation of new scientific evidence in real-world practice settings [1–4]. There is evidence to suggest that stakeholder engagement is the key in both implementation and translational sciences which includes tailoring best practices for specific populations [5–7]. Although there has been a proliferation of work on stakeholder engagement, there is limited empirical evidence on the best practices for stakeholder engagement and even less on evaluation of engagement demonstrating the association between the quality and quantity of engagement and research outcomes. However, there is a critical difference between going through the empty ritual of obtaining stakeholder feedback and giving stakeholders the real power needed to affect the research process and resulting outcomes [8].

CLASSIFICATION OF STAKEHOLDER ENGAGEMENT

There is a broad spectrum of stakeholder engagement, but we believe engagement falls primarily within three broad categories: (1) non-participation, (2) symbolic participation, and (3) engaged participation with each of these categories containing subcategories on the engagement continuum (see Fig. 1). While some people may consider work in the non-participation category stakeholder engagement, we do not support this as genuine stakeholder engagement, as the usual purpose is not to engage stakeholders in the planning, implementation, evaluation, and decision-making, rather in these cases powerful institutions are often trying to educate or treat participants. The next level symbolic participation affords stakeholders a place at the table allowing them to hear plans and have a voice. The question is whether community health stakeholder voices are heard and carry weight or do the voices of academic researchers overpower those of the community. When stakeholder engagement is in this category, there are no assurances that there will be changes in the status quo. If the stakeholders are allowed to advise, but the researchers have the ultimate decision-making power, in essence, the presence of stakeholders gives the appearance of engagement, but they are not meaningfully engaged which includes shared decision-making. In the engaged participation category community health stakeholder including patients, caregivers, and advocacy groups that traditionally have limited power are given shared decision-making authority with powerful stakeholders (e.g., hospital systems, health center leadership, and academic researchers), and these stakeholders collaboratively manage the project based on stakeholder priorities [8].

Fig. 1.

Categories and classifications of stakeholder engagement. Stakeholder engagement is grouped into three broad categories: non-participation, symbolic participation, and engaged participation. Each of the broad categories is classified into subgroups. Subgroups in non-participation include outreach and education. Subgroups in symbolic participation include coordination and cooperation. Subgroups in engaged participation include collaboration, patient-centered, and community-based participatory research (CBPR)

It is important to state that Fig. 1 is a simplification of the partnership spectrum and cannot illustrate the nuanced differences among categories that many may miss at first glance. These differences may seem slight, but it is important to determine where a project falls on the stakeholder engagement spectrum, as this can have implications for sustainability, effectiveness, and outcomes. We propose the following definitions for stakeholder engagement classifications.

Outreach: Researchers develop, implement, and evaluate strategies to reach the target population. Key members of the target population (gatekeepers) can be engaged as advisors and make key connections.

Education: Researchers are trying to educate stakeholders about a particular topic. This is usually combined with outreach efforts to gain audiences for education sessions and/or materials.

Coordination: Researchers gather community health stakeholders together to assess important elements of a project or activity. Community members give feedback, and this feedback informs researchers’ decisions, but it is the researchers’ responsibility to design and implement the study with no help expected from the community members. Research and related programs are strengthened through community outreach, and results are disseminated through community groups and gatekeepers.

Cooperation: Researchers ask community members for help with a project, instead of just asking for advice. There is some activity on the part of community members in defined aspects of the project, including recruitment, implementation of interventions, measurement, and interpretation of outcomes. Community health stakeholders are ongoing partners in the decision-making for the project. Community health stakeholder understanding of research and its potential importance is enhanced through participation in activities.

Collaboration: Both researchers and community members are actively involved in the design and implementation of the project and the interpretation of the findings. In addition, all stakeholders benefit in some way from working together including increased capacity of community groups to engage in research implementation. Community health stakeholders collaborate in decision-making and resource allocation with an equitable balance of power that values input from the community health stakeholders.

Patient-centered: Patients, caregivers, and advocacy groups dictate the priority setting for research choices and control the design and implementation of the project activities, and the interpretation and publication of findings. Researchers use their expertise to move these things along, but community health stakeholders make all major decisions about research approaches. There are systems in place for patient participation in research at all engagement levels. Community health stakeholders have the capacity to engage in partnerships with an equitable balance of power for governance and a strong level of accountability to the public/community.

Community-based participatory research: CBPR is the population health approach to the patient-centered engagement model. The principles of community-based participatory research highlight trust among partners, respect for each partner’s expertise and contributions, mutual benefit among all partners, and a community-driven partnership with equitable and shared decision-making [9–12].

Community health stakeholders are a heterogeneous group with a variety of perspectives, needs, resources, skills, and interests. People do not always fit in nice categorical groupings and do not necessarily represent the perspectives or interests of the groups they may be perceived to fit in. Within each man-made categorization, there are divergent perspectives, competing interests, and the potential for splintered subgroups. Despite this natural heterogeneity, people in one group often think of those in other groups as homogeneous; “in most cases the have-nots really do perceive the powerful as a monolithic ‘system’, and powerholders actually do view the have-nots as a sea of ‘those people’, with little comprehension of the class and caste differences among them.” [8] Therefore, it is important to consider all community health stakeholders when determining who gets a seat at the table and to value the diversity of perspectives among those at the table.

IMPLEMENTATION OF STAKEHOLDER ENGAGEMENT

As evidence-based quality improvement is an iterative strategy requiring alternating action and reflection, it is important for all stakeholders to understand systems and the problems they create, while developing and evaluating solutions to these concerns. In this regard, community-based participatory research (CBPR) articulates principles that may contribute to processes that yield stronger improvement projects and outcome data. While there are other principles identified and used, generally, CBPR efforts are believed to achieve their aims through a focus on bringing together multiple stakeholders, including practitioners, researchers, and communities to establish trust, share power, enhance strengths and resources, and examine and address needs and health problems with solutions developed in collaboration [13, 14].

A key aspect of CBPR is the engagement of appropriate stakeholders, in all phases of the work and in a collaborative process that recognizes the strengths and assets of partners. Hamilton et al. discuss the process used for stakeholder engagement in evidence-based quality improvement research [15]; however, there are aspects of the process that are unclear and other components that appear to be more limited in implementation than is optimal, with implementation falling somewhere between cooperation and collaboration in our classification scheme. As demonstrated in the work of Hamilton et al., there are four key constructs to consider when engaging multilevel stakeholders in the research process: (1) commitment of stakeholders to the process and the goals of the project; (2) capacity of stakeholders to participate in the process and engage in research activities; (3) commitment of researchers to meaningfully engaged stakeholders; and (4) trust among researchers and stakeholders [15].

Planning processes are important for any activities that seek engagement of multiple stakeholders to identify and address changes required to improve system effectiveness and efficiency. For example, it is important to have a phase that allows partners to get to know each other, understand each unique perspective on the issue to be considered, and how their perspective allows them to contribute to the effort. The Hamilton study suggests that there was some effort made in this regard, but that it was perhaps insufficient for the variety of teams formed. It is also important that project outcomes are determined on the basis of the partnership purpose, with the data informing outcomes reviewed by all partners. Stakeholders must be able to access and review the data with sufficient time to critique and perhaps add to the data, as well discuss project purpose. We found it difficult to evaluate the process used by Hamilton et al. given the brief discussion on this topic [15].

As the planning process continues, participants should determine the resources required to complete each activity. Time is a critical resource in the health care environment and acknowledgment, and planning for its allocation to evidence-based quality improvement (EBQI) is critical. The ability of the team to obtain needed resources for the intervention can help to determine if activities and the expected outcomes are realistic. As activities are decided and refined, the following should also be determined: the responsibilities of each partner, the timeline, resources provided, received, and shared. Logic models may be helpful to teams to assist in the process of developing and documenting their objectives, inputs, activities, outcomes, and timeline. As the partnership changes, expands or contracts, activities will shift, as will other elements of the logic model. Ongoing communication and opportunities for discussion are important to managing project changes. Because the process is iterative and subject to change, flexibility is required for all stakeholders. Hamilton et al. do not fully describe how resources, such as time, were negotiated for and among EBQI teams [15].

Optimally, the schedule and plan for reviews of project milestones are discussed and developed during the planning process. It was unclear from the details provided whether the implementation processes followed by Hamilton et al. included levels of training and technical assistance sufficient to assure sustained communication, decision-making, shared vision, and change management fundamental to an engaged processes. What was clear was that some teams functioned well and others struggled. The level of detail provided only allows us to speculate on the reasons for the differences [15]. It is possible that without clear goals, assignment of responsibility and periodic communication, interest, and effort may have waned. This state of affairs would not be unique to EBQI, as this is a common lesson learned from stakeholder-engaged research. In addition, the discussion of outcomes should have included consideration for how the selected outcomes and activities served the patient population, further highlighting the importance of their inclusion among key stakeholders [14]. Patient inclusion may have resulted in goals that were valued sufficiently to retain interest over time. It is unlikely that any deficiencies noted and discussed as potential issues for the teams studied are significantly different than those observed in other stakeholder-engaged research, but this is exactly the type of information that is needed to assist in the development of the science of stakeholder engagement.

We commend the authors for their use of evidence-based data driven approaches to develop shared priorities with the understanding that targeted initiatives work better than global initiatives [15]. However, we encourage them and others to take this work further to include meaningful engagement of patients (or the community/population impacted by the proposed work), including efforts for the development of processes for continued communication, decision-making, shared vision, change management, and implementation (at initiation), which are the key elements of an engaged processes. In addition, there is the need to assess stakeholder engagement in the process and examine the association between the quality and quantity of engagement and the resulting outcomes. This will allow for the identification of best practices in specific settings.

Despite years of interest in CBPR, its adoption and implementation has been variable. A systematic review of CBPR clinical trials showed that most CBPR studies reported that involvement varied by level and type of activities, such as identifying study questions, recruitment efforts, development and delivery of the intervention, and data collection methods [16]. We suggest that those interested in engaging multiple stakeholders start with capacity building efforts for community health stakeholders. In our work in St. Louis, we have developed, implemented, and evaluated a training program for increasing research literacy among community health stakeholders [17–19]. This initial commitment by researchers can enhance the infrastructure for partnership development and ensure that community health stakeholders have the research capacity for meaningful and sustained engagement.

EVALUATION OF STAKEHOLDER ENGAGEMENT

There is recognition that stronger evaluations are needed to refine the patient-centered medical home model and to maximize their effectiveness and efficiency [13, 20]. In addition, data are needed to understand the effects of efforts to improve practices in targeted areas [20]. The major goal of these efforts is to ensure improved patient care through a focus on principles that include personal preferences, a focus on the whole person, coordinated and/or integrated care, and quality and safety improvement [13]. Although primary care quality improvement teams have been implemented in some settings, there has been minimal evaluation of these efforts [21]. For this reason, the work of Hamilton et al. makes an important contribution to the literature [15]. Although Hamilton et al. made an admirable effort to understand and explicate implementation of evidence-based quality improvement for gender-tailored patient-centered medical homes, it is not clear that sufficient attention was paid to strategies to assure inclusion of all stakeholders, co-learning and optimizing community benefit [15].

With the consideration for limited word counts allotted for journal articles, it would be important to understand what elements of goal setting and planning took place among the EBQI teams studied. In program/project development and planning, the standard criteria for outcomes are that they be specific, measurable, achievable, realistic, and timely (SMART) [22]. The use of specific and measurable goals facilitates review and evaluation of partnership activity and is important to understanding partnership progress and the need for changes in activities [14]. However, the process by which SMART goals are developed by a team matters. Does perceived expertise affect level of participation in goal setting and other aspects of the team process? As the partnership meets goals, expands, or shifts interests, or the needs of the target population change, partnership purpose, and outcomes can be reviewed and revised. This process can be important in maintaining stakeholder interest as our own partnership work focused on community-based cancer prevention has demonstrated, but planning for this level of review and revision is difficult [23].

In our stakeholder engagement evaluation efforts, we were trying to elucidate best practices for community engagement in Community Network Program Centers addressing cancer disparities but encountered a major stumbling block, as we found a lack of quantitative measures that assess the level of community engagement in research from the stakeholder perspective [23, 24]. This led us to develop a comprehensive quantitative measure [25]; the focus of our current work is on validating this measure and studying the implementation of the measure, as it is used in practice. We encourage others to share measures they are using in practice and their best practices for and lessons learned from using stakeholder-engaged evaluation approaches.

THE SCIENCE OF STAKEHOLDER ENGAGEMENT

While the utility of stakeholder engagement in research has been demonstrated [12, 14, 26–28], we need to begin to think about the science of stakeholder engagement in research. How do we measure the level of engagement and the impact of that engagement on research outcomes? What are the best practices for implementation of multilevel stakeholder engagement? How do we evaluate which approaches are most effective in different settings?

Successful stakeholder engagement in an individual project is commendable. However, the science of community engagement requires that we move beyond individual projects to a broader understanding of what works and why (best practices). As with any science, the future is dependent on the foundation that it is built upon. Thus, as we engage multilevel stakeholders in our work, it is imperative to evaluate both the process and outcomes of this work. The key to this evaluation is the perspective of stakeholders regarding the benefit and outcomes of the project and recognition of the unique cultural traits and geographic settings of the target population. However, given the nature of partnerships, this effort is not one-sided, as the institutional culture and perception of institutions among the target population are also important to consider.

When implementing and evaluating stakeholder engagement in research, some key questions to consider are the following:

• Who will benefit from this work?

• Which are the appropriate stakeholders to engage to address a problem?

• Where is your partnership on the stakeholder engagement continuum?

• Where would you like your partnership to be on the stakeholder engagement continuum?

• What processes should be developed and used for partnership sustainability and progress along the stakeholder engagement continuum?

• If a group of stakeholders participate and only one benefits, was the engagement meaningful or perfunctory?

• How will you evaluate the quality and quantity of stakeholder engagement?

• Does stakeholder engagement improve the fit of evidence-based practices in community-based settings?

With these questions in mind, project teams can prepare for the challenging task of meaningful engaged participation from community health stakeholders. Successful partnerships are developed and sustained when the constituent members contribute their perspectives, resources, and skills creating an amalgam for research synergy allowing the partnership to obtain outcomes that no one constituent member could have produced on their own. We acknowledge that what we are proposing is not easy, but the benefits can be impactful if we speed implementation and dissemination by continually sharing what works providing a foundation for others.

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.

Funding

The stakeholder engagement classification work is supported through a Patient-Centered Outcomes Research Institute (PCORI) award (ME-1511-33027). All statements in this report, including its findings and conclusions, are solely those of the authors and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute (PCORI), its Board of Governors, or Methodology Committee.