Abstract
Introduction:
A demonstration project in Richmond, Virginia involved patients and other stakeholders in the creation of a research agenda on dietary and behavioral management of diabetes and hypertension. Given the impact of these diseases on morbidity and mortality, considerable research has been directed at the challenges patients face in chronic disease management. The continuing need to understand disparities and find evidence-based interventions to improve outcomes has been fruitful, but disparities and unmet needs persist.
Methods:
The Stakeholder Engagement in Question Development (SEED) method is a stakeholder engagement methodology that combines engagement with a review of available evidence to generate research questions that address current research gaps and are important to patients and other stakeholders. Using the SEED method, patients and other stakeholders participated in research question development through a combination of collaborative, participatory, and consultative engagement. Steps in the process included: (1) identifying the topic and recruiting participants; (2) conducting focus groups and interviews; (3) developing conceptual models; (4) developing research questions; and (5) prioritizing research questions.
Results:
Stakeholders were involved in the SEED process from February to August 2015. Eighteen questions were prioritized for inclusion in the research agenda, covering diverse domains, from healthcare provision to social and environmental factors. Data analysis took place September to May 2016. During this time, researchers conducted a literature review to target research gaps.
Conclusions:
The stakeholder-prioritized, novel research questions developed through the SEED process can directly inform future research and guide the development of evidence that translates more directly to clinical practice.
INTRODUCTION
Ademonstration project in Richmond, Virginia was conducted to involve patients and other stakeholders in the creation of a research agenda on dietary and behavioral management of diabetes and hypertension. The study tested a new stakeholder engagement methodology, the Stakeholder Engagement in Question Development (SEED) method. SEED combines stakeholder engagement with a review of the evidence base to generate research questions important to patients and other stakeholders that address gaps in the available research evidence.
The Centers for Disease Control and Prevention estimates that 29.1 million Americans had diabetes in 2014. Among adults, age-adjusted diabetes rates are highest among Native Americans/Alaska Natives (15.9%), non-Hispanic blacks (13.2%), and Hispanics (12.8%). Poorly controlled diabetes greatly increases the risk of serious complications, including heart disease and stroke, retinopathy and blindness, kidney disease, peripheral neuropathy, and amputations.1 It was the seventh leading cause of death in 2013, with an age-adjusted death rate of 21.2 per 100,000.2 Hypertension affects approximately 70 million American adults, about half of whom have poorly controlled blood pressure, putting them at risk for heart disease, stroke, and hypertensive renal disease.3 Hypertension rates in the U.S. are highest among non-Hispanic black adults.4 Essential hypertension/hypertensive renal disease was the 13th leading cause of death, with an age-adjusted death rate of 8.5 per 100,000.2 Diabetes and hypertension management includes appropriate diet, exercise, and medications.
Given the impact of these chronic illnesses on morbidity and mortality, research is needed that addresses the challenges patients face in chronic disease management. The continuing need to understand disparities and find evidence-based interventions to improve outcomes has been fruitful, but disparities persist—perhaps because investigators have not fully understood the challenges faced by patients. A great deal of research on these conditions can be expected in the future and methods for successfully engaging stakeholders in shaping that agenda may result in studies that lead to more actionable solutions.
Patients and stakeholders, based on their lived experience, are in a unique position to pose hypotheses about how health-related factors are inter-related and propose intriguing questions for empirical explorations that reflect their needs and priorities. Involving patients and other stakeholders in health research may provide opportunities to produce research that is more valid, relevant, accepted, sustainable, and innovative.5–10 Research questions developed through stakeholder engagement have the potential to generate research evidence that translates more directly to inform clinical practice. However, there are limitations to existing approaches to stakeholder engagement,11 including engaging lay people in the prioritization of research questions. For example, lay people may be under-represented on research teams, or asked to make decisions about unfamiliar technical information,12 raising concerns about the extent to which their contributions can rise above the implicit power dynamics of research development. Participatory research teams appear to have had greater success.13 A systematic review of studies employing stakeholder participation found advantages to strategies that used a collaborative process; combined collaboration with consultative engagement; engaged stakeholders directly and repeatedly; engaged well-networked stakeholders; and provided stakeholders with information, resources, and support.14
For those interested in the collaborative generation of research questions, there are few comprehensive research frameworks available.10 Previous studies have successfully gathered data on stakeholder preferences for research through consultative methods, such as focus groups,15 but lack a collaborative framework for systematically engaging stakeholders and for capacity building. The SEED method is a collaborative process designed to provide meaningful engagement of stakeholders in the research process, particularly those in hard-to-reach populations. This demonstration engaged stakeholders over 9 months in a process of gathering and reviewing data, gaining perspectives from other stakeholders, and concept mapping, prior to generating and prioritizing research questions. Concept mapping allows stakeholders to explore the topic in depth and produce conceptual diagrams that can serve as the basis for connecting the theoretic framework to empirical data, denoting the direction of causality, linking variables temporally, and mapping interactions between variables.
METHODS
Previous research has pointed to the importance of using a combination of collaboration and consultation, “with members of the public taking a leading role in consulting their peers.”10 The SEED method employs a multilevel stakeholder model that is led within a participatory framework. Engagement encompasses three levels of participation: The first level is a collaborative research team with lay participants and researchers who are involved in the project from beginning to end.16 The second level engages patients and other stakeholders selected and recruited by the research team in “topic groups” to develop the research questions. The third level brings in broader consultative stakeholder participation in the form of focus groups and interviews.
The method follows a five-stage process (Figure 1). In Stage 1 (identify and engage), the research team identified the topic and prioritized and recruited participants. The research team in this demonstration was Engaging Richmond, an existing community–university partnership. The topic of diet compliance for diabetes and hypertension was selected through a collaborative process by Engaging Richmond along with Virginia Commonwealth University’s practice-based research network. The topic scope was later expanded to include behavior management based on the actual questions posed by the stakeholders. The research team reviewed health statistics and demographic data and conducted ten informational interviews with local healthcare system representatives to learn about the at-risk population and where they receive services in the community.
Figure 1. Steps in the SEED Method.
SEED, Stakeholder Engagement in Question Development.
The selection of topic stakeholders was guided by the health issue and the groups affected by it. The research team completed a set of “stakeholder identification and recruitment matrices” developed for the SEED method. The tool guided the research team through a participatory process of listing patient subgroups in the community (e.g., by gender, age, race, health status, and others), agreeing on a set of selection criteria, and prioritizing groups based on those criteria. The criteria used to rank each group were: (1) level of risk related to the health problem; (2) presence in the study area; and (3) vulnerability. Groups that ranked high across all three criteria were selected as potential topic group participants. The team then selected other stakeholders (e.g., health professions) using a similar matrix. Using this process, the research team recruited participants for three topic groups: (1) seniors with diabetes or hypertension, particularly those with comorbid conditions; (2) adults with diabetes or hypertension and limited healthcare access or other social or economic challenges; and (3) nurses, health educators and other service providers. The recruitment matrices were used to develop a plan for community venues and organizations to reach out to for help with recruitment. The research team recruited participants using flyers distributed to selected resource centers, community centers, and healthcare offices and employed targeted recruitment strategies through team members’ professional networks. Among all participants, about three-quarters were African American. About half of participants had a high school education or less, while half had some college or more education. All participants received hourly payment (research team) or a stipend (topic groups and interview/focus group participants). Human subjects participation was approved by Virginia Commonwealth University’s IRB.
While the topic group participants were involved in the project over several months, in Stage 2, (consult) additional stakeholders engaged in focus groups and interviews intended to broaden topic group and research team members’ understanding of the experiences of a diverse set of stakeholders. Five focus groups were conducted with female African Americans, seniors, food pantry clients, Supplemental Nutrition Assistance Program recipients, and people taking medications for diabetes or hypertension. Eleven one-on-one interviews were conducted with healthcare workers, service providers, and parents of children with diabetes. During this stage, the topic groups became familiar with the project and reviewed disease information and focus group and interview findings.
In Stage 3 (conceptualize), the topic groups each developed conceptual models that explored potential causal factors leading to management of diabetes or hypertension. Causal pathway diagrams are widely used as an analytic framework for proposing the interrelationships between dependent and independent variables and mediating and moderating factors. These models can be useful in making assumptions explicit; providing a framework for data analysis; generating testable predictions; exploring the effects of interventions; identifying data gaps17; guiding the formulation of research questions, study design, analysis, and interpretation18; directing future research19; or helping develop interventions.20 The Center on Society and Health had piloted the use of participatory conceptual modeling in several projects before they were incorporated into the SEED method.21,22
Topic groups participated in training on conceptual modeling and in the next meeting engaged in the conceptual modeling activity facilitated by the principal investigator (PI). Topic group members first worked individually to identify factors related to the outcome and then reviewed a worksheet of factors to help expand their lists. The groups discussed all selected factors and categorized them into different domains (e.g., environmental, demographics). Each group then utilized the factors to collaboratively build a unique conceptual model while discussing the significance of each factor and its relationship to the other factors in the model. A white board and sticky notes and arrows were used so that items could be moved interactively as the model building progressed. The PI then created graphic versions of each model, which were edited by the research team to reflect the models created by the participants. The topic groups also reviewed and edited the final models and compared models across groups (Figure 2).
Figure 2. Sample conceptual model created by a Topic group.
SNAP, Supplemental Nutrition Assistance Program.
In Stage 4 (generate questions), the topic groups received a brief training on creating research questions and participated in an activity facilitated by the PI to develop questions. The facilitator provided a series of specific prompts to encourage creation of questions focused on causes (Which pathways are critical to understanding the outcome?), impact (Which pathways are most likely to respond to intervention?), patient-centeredness (What would help patients make more-informed decisions?), verification (Are there promising relationships we need to know more about?), and new directions (What is a new way of thinking about this?). The groups engaged in discussion about each question as it was presented.
In Stage 5 (prioritize questions), topic groups reviewed and further discussed each question and prioritized the top questions through a voting procedure. Once each topic group had a final set of questions, the facilitator led another discussion to probe participants’ rationale for presenting each question and dig deeper into issues related to patient-centeredness, populations of interest, outcome measures, study timeframes, and study settings. Afterward, the research team reviewed and discussed each question to explore different perspectives and provided additional context related to each question.
In the last step (finalize and disseminate), a review team (including the PI, faculty and staff, and graduate students) was formed to research and finalize the questions by conducting a literature review. The review investigated which aspects of each question had been answered by prior studies and made recommendations targeting research gaps. An example of the process is Question 7 (Appendix Table 1, available online): Will controlling diabetes and hypertension prevent other diseases or more serious illness? There is definitive research evidence on the many complications related to uncontrolled diabetes and hypertension. Therefore, the review focused on the patients’ need for more information related to complications—an issue that was discussed repeatedly during topic group meetings and focus groups. The review team then made recommendations to address relevant gaps in the peer-reviewed literature. Research and subject matter experts were consulted on each finalized question for feedback on the wording of the recommendations and to ensure their relevance to their respective fields of study. Dissemination of the results is ongoing. Findings have been disseminated to community stakeholders and local research groups, and are being shared with funding agencies and national advocacy groups. Presentations at several national conferences have highlighted both the method and the preliminary findings and have included study participants from the research team and topic groups.
RESULTS
Appendix Table 1 (available online) presents a stakeholder-created research agenda that should appeal to researchers, funders, and advocates across disciplinary areas and organizational types. Sixty-two questions were presented by the topic groups and 19 were prioritized for inclusion in the research agenda, covering diverse domains from healthcare provision to social and environmental factors. One question was subsequently dropped because it did not relate directly to the topic of diet and behavior management. Each question is presented as posed by the stakeholders, with minor wording revisions as needed for clarity, followed by a list of recommendations based on potential research gaps identified through the literature review. Some recommendations relate to specific subpopulations, particularly those of interest to the participants or populations for which there is a lack of research. The recommended topics are not worded as final research questions in anticipation that researchers will craft specific questions based on their hypotheses and study approaches. These topics are presented broadly, calling attention to potentially relevant gaps in research knowledge, with the aim of heightening their applicability across diverse contexts and settings.
Prior studies of stakeholder involvement in creating research questions have rarely looked systematically at how input varies across different types of stakeholders, and scarce data are available on the differences in research priorities among stakeholder groups.23 Some studies have noted the unique contributions of lay people to question development, such as a project on traumatic brain injury that found some research questions that were ultimately designated as high priority “were topics that clinicians and researchers had not thought of themselves.”24 To compare the contributions of the three groups of topic stakeholders, each of the prioritized questions was coded by subject, revealing a fair amount of overlap in the subjects proposed by the three groups, while each group also had unique contributions. For example, the senior group was the only one whose questions related to physical/mental health, healthcare economics, and healthcare treatment. On the other hand, all three topic groups prioritized questions related to healthcare quality, health communication, and policy. Further analysis of differences between stakeholders will be presented with the demonstration evaluation report.
The authors collected feedback on the process throughout the demonstration period via surveys and interviews. The research team’s feedback expressed the centrality of an engaged group process, identifying five facilitative themes that included investing time at each stage of the process to ensure all members of the team are authentically engaged; being thorough in breaking down all steps; encouraging participants to demonstrate persistence by asking thorough questions; showing respect by minimizing sidebar conversations and respecting leadership roles that emerged within and among group members; and managing conflict through open, frequent reminders of the core values of the group. The evaluation also highlighted the need to address uncertainties about the process with participants through tangible reminders and ongoing education combining visual, spoken, and action-based orientation to each step of the process. Evaluation findings pointed to the need to pay attention to the importance of conceptual modeling as a crucial step in the research process that may be unfamiliar to those without a research background and to devote more time to it. Overall, feedback indicated that the SEED method contributed to participants’ confidence, leadership skills, and awareness of team dynamics, which would have direct application to their individual development while advancing the quality of community-engaged research.
DISCUSSION
Patient-centeredness is achieved by shaping research questions around the concerns, needs, and values of stakeholders.24 The results presented here derive from a demonstration with a small and select group of stakeholders—the research agenda is presented to explore stakeholder perspectives and open opportunities for discussion and inquiry rather than propose a definitive set of questions or priorities. Diverse viewpoints and experiences can bring a greater level of understanding to the processes by which social and environmental factors affect health-related behaviors, decision making, and health outcomes. Patients and other stakeholders bring experiential knowledge to the research endeavor that may differ in important ways from that of medical and health science researchers,2,25 and their research priorities may differ based on different conceptions of what is important to learn15,25–27 or contribute to new innovations28 and potentially improve research translation.
Interest in stakeholder-developed research agendas is growing. A number of articles have presented recommended research agendas on specific health topics.29–31 Some of these efforts, such as those sponsored by funders and interest groups, have a built-in audience that may be predisposed to initiate or fund projects resulting from these efforts. Independent projects such as this one have no such built-in audience and their potential impact is currently unknown. However, interest in the multitude of factors that shape health outcomes, including social determinants of health, is growing. For example, a 2015 Kaiser Family Foundation issue brief calls attention to emerging efforts to integrate social and environmental needs into the healthcare system, including efforts by states, Medicaid, providers, and health plans.32 Stakeholder-developed research agendas can help make the case for decisions about prioritizing future research projects and help bolster support for funding proposals. This demonstration was meant to create a research agenda that generates interest among diverse audiences in funding or pursing research on topics relevant to patients and others stakeholders. The methodology may be adapted by future users, for example, to create and prioritize questions focused on interventions or policy. Early feedback on the SEED method has garnered interest in using it as a next stage in existing community engagement initiatives, providing a participatory framework for developing and prioritizing research ideas.
CONCLUSIONS
This is the first article to report the findings from this demonstration of the SEED method. A second demonstration is currently underway in rural Virginia on the topic of respiratory cancer outcomes. Articles that provide results from that demonstration, as well as more detail on the methodology (including a toolkit) and the findings of the process and outcome evaluations will follow. Further demonstrations are planned to test alternative implementation strategies. In addition, the research team will gather evidence about the impact of the research agendas generated by these demonstrations, tracking the responses of target audiences, how they envision using the information, and whether any of these questions impact funding or research decisions. An over-arching goal of the SEED method is that the needs and priorities of stakeholders that have driven these questions will provide impetus and support to addressing these questions. A longer-term goal is to encourage others to develop stakeholder research agendas using this methodology and to develop a repository of such efforts.
Supplementary Material
ACKNOWLEDGMENTS
Thank you to the following individuals for providing feedback on specific questions in this research agenda: Danielle Berman, PhD (U.S. Department of Agriculture), Thomas DeLeire, PhD (Georgetown University), Rebecca Etz, PhD (Virginia Commonwealth University), Jag Khalsa, PhD (National Institute on Drug Abuse), Kirsten Knutson, PhD (Northwestern University), Kathryn Kolasa, PhD (East Carolina University), Alex Krist, MD, MPH (Virginia Commonwealth University), Anton Kuzel, MD, MHPE (Virginia Commonwealth University), Pearl Lee, MD (University of Michigan), Theresa J. Nartea, DBA (Virginia State University), Gretchen Piatt, PhD (University of Michigan), Hilary Seligman, MD (University of California, San Francisco), Joni Strom Williams, MD, MPH (Medical University of South Carolina). The following students and staff at the Virginia Commonwealth University Center on Society and Health helped make the research and manuscript possible: Josh Brown, Jennifer Early, Jill Hellman, Julie Karr, Megan Kervin, Isra Malik, and Albert Walker. Engaging Richmond team members who helped lead this project were Sheila Goode, Danita Gregory, Sharon Herman, Brenda Kenney, Chimere Miles, Audrey Smith, and Travis White. Finally, the authors have deep appreciation for the participation of those stakeholders who were members of topic groups and those who took part in focus groups and interviews.
Research reported in this article was funded through a Patient-Centered Outcomes Research Institute (PCORI) Award (ME-1310-07664). The statements in this article are solely the responsibility of the authors and do not necessarily represent the views of PCORI, its Board of Governors, or Methodology Committee. PCORI had no role in study design; collection, analysis, and interpretation of data; writing the report; or the decision to submit the report for publication.
Tasks completed by the authors are as follows: EBZ developed the study design, supervised the study, facilitated meetings, analyzed results, performed the literature review, and developed and edited the manuscript. SC coordinated study implementation, collected and analyzed data, performed the literature review, and reviewed the manuscript. ADH facilitated project meetings, performed the literature review, and reviewed the manuscript. SHW consulted on study design and implementation, and edited the manuscript. SKP consulted on study design and implementation, and edited the manuscript.
The study findings have not been previously published elsewhere. An overview of the study methods and the Richmond City demonstration project were presented at several conferences (South Carolina Public Health Association 2015; American Public Health Association 2015; Campus Community Partnerships for Health 2016; American Sociological Association August 2016 and Advancing the Science of Community Engaged Research conference August 2016). Posters on the project were presented at Association for Clinical and Translational Science Conference 2016 (focused on evaluation results) and Academy Health 2016 (focused on stakeholder contributions, including a brief summary of stakeholder questions).
No financial disclosures were reported by the authors of this paper.
Footnotes
SUPPLEMENTAL MATERIAL
Supplemental materials associated with this article can be found in the online version at http://dx.doi.org/10.1016/j.amepre.2017.01.034.
REFERENCES
- 1.CDC . National Diabetes Statistics Report: Estimates of Diabetes and Its Burden in the United States. Atlanta, GA: U.S. DHHS, 2014. [Google Scholar]
- 2.Xu J, Murphy SL, Kochanek KD, Bastian BA. Deaths: final data for 2013. Natl Vital Stat Rep. 2016;64(2). [PubMed] [Google Scholar]
- 3.CDC. High blood pressure fact sheet. www.cdc.gov/dhdsp/data_statistics/fact_sheets/docs/fs_bloodpressure.pdf. Accessed February 22, 2016.
- 4.Nwankwo T, Yoon SS, Burt V, Gu Q. Hypertension among adults in the United States: National Health and Nutrition Examination Survey, 2011–2012. NCHS Data Brief. 2013(133):1–8. [PubMed] [Google Scholar]
- 5.Entwistle VA, Renfrew MJ, Yearley S, Forrester J, Lamont T. Lay perspectives: advantages for health research. BMJ. 1998;316 (7129):463–466. 10.1136/bmj.316.7129.463. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Winder E. The Value+ toolkit. European patient’s forum. www.eu-patient.eu/globalassets/projects/valueplus/value-toolkit.pdf. Accessed July 13, 2016. [Google Scholar]
- 7.Caron-Flinterman JF, Broerse JEW, Bunders JFG. The experiential knowledge of patients: a new resource for biomedical research? Soc Sci Med. 2005;60(11):2575–2584. 10.1016/j.socscimed.2004.11.023. [DOI] [PubMed] [Google Scholar]
- 8.Goodare H, Lockwood S. Involving patients in clinical research: improves the quality of research. BMJ. 1999;319(7212):724–725. 10.1136/bmj.319.7212.724. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Domecq Garces JP. Eliciting Patient Perspective in Patient-Centered Outcomes Research: A Meta Narrative Systematic Review A report prepared for the Patient-Centered Outcomes Research Institute. Rochester, MN: Mayo Clinic; 2012. [Google Scholar]
- 10.Nass P, Levine S, Yancy C. Methods for involving patients in topic generation for patient-centered comparative effectiveness research: an international perspective. Research Priorities White Paper (PCORI-SOLRPWG-001) for the Patient-Centered Outcomes Research Institute (PCORI). www.pcori.org/assets/Methods-for-Involving-Patients-in-Topic-Generation-for-Patient-Centered-Comparative-Effectiveness-Research-%E2%80%93-An-International-Perspective.pdf. Accessed July 13, 2016. [Google Scholar]
- 11.O’Haire C, McPheeters M, Nakamoto EK, et al. Engaging Stakeholders to Identify and Prioritize Future Research Needs. Methods Future Research Needs Reports, No. 4 Rockville, MD: Agency for Healthcare Research and Quality; 2011. [PubMed] [Google Scholar]
- 12.Hoffman A, Montgomery R, Aubry W, Tunis SR. How best to engage patients, doctors, and other stakeholders in designing comparative effectiveness studies. Health Aff (Millwood). 2010;29(10):1834–1841. 10.1377/hlthaff.2010.0675. [DOI] [PubMed] [Google Scholar]
- 13.Wright D, Corner J, Hopkinson J, Foster C. Listening to the views of people affected by cancer about cancer research: an example of participatory research in setting the cancer research agenda. Health Expect. 2006;9(1): 3–12. 10.1111/j.1369-7625.2006.00353.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Oliver S, Clarke-Jones L, Rees R, et al. Involving consumers in research and development agenda setting for the NHS: developing an evidence-based approach. Health Technol Assess. 2004;8(15):1–148. 10.1017/S026646230400073X. [DOI] [PubMed] [Google Scholar]
- 15.Welfare MR, Colligan J, Molyneux S, Pearson P, Barton R. The identification of topics for research that are important to people with ulcerative colitis. Eur J Gastroenterol Hepatol. 2006;18(9):939–944. 10.1097/01.meg.0000230088.91415.5b. [DOI] [PubMed] [Google Scholar]
- 16.Minogue V, Boness J, Brown A, Girdlestone J. The impact of service user involvement in research. Int J Health Care Qual Assur. 2005;18 (2):103–112. 10.1108/09526860510588133. [DOI] [PubMed] [Google Scholar]
- 17.Joffe M, Mindell J. Complex causal process diagrams for analyzing the health impacts of policy interventions. Am J Public Health. 2006;96 (3):473–479. 10.2105/AJPH.2005.063693. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Diez Roux AV. Conceptual approaches to the study of health disparities. Annu Rev Public Health. 2012;33:41–58. 10.1146/annurev-publhealth-031811-124534. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Paradies Y, Stevens M. Conceptual diagrams in public health research. J Epidemiol Community Health. 2005;59(12):1012–1013. 10.1136/jech.2005.036913. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Schulz AJ, Parker EA, Israel BA, Allen A, Decarlo M, Lockett M. Addressing social determinants of health through community-based participatory research: the East Side Village Health Worker Partnership. Health Educ Behav. 2002;29(3):326–341. 10.1177/109019810202900305. [DOI] [PubMed] [Google Scholar]
- 21.Woolf SH, Braveman P, Evans BF. The Health Implications of Reduced Food Stamp Eligibility. Richmond, VA: Virginia Commonwealth University, Center on Human Needs; 2013. [Google Scholar]
- 22.Zimmerman EB, Woolf SH, Haley A. Understanding the relationship between education and health: a review of the evidence and an examination of community perspectives In: Kaplan RM, Spittel ML, David DH, eds. Population Health: Behavioral and Social Science Insights. Rockville, MD: Agency for Healthcare Research and Quality; 2015:347–384. [Google Scholar]
- 23.Owens C, Ley A, Aitken P. Do different stakeholder groups share mental health research priorities? A four-arm Delphi study. Health Expect. 2008;11(4):418–431. 10.1111/j.1369-7625.2008.00492.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Clavisia O, Braggea P, Tavendera E, Turnerb T, Gruena RL. Effective stakeholder participation in setting research priorities using a Global Evidence Mapping approach. J Clin Epidemiol. 2013;66(5):496–502. 10.1016/j.jclinepi.2012.04.002. [DOI] [PubMed] [Google Scholar]
- 25.Corburn J. Community knowledge in environmental health science: co-producing policy expertise. Environ Sci Policy. 2007;10(2):150–161. 10.1016/j.envsci.2006.09.004. [DOI] [Google Scholar]
- 26.Mcintyre S, Novak I, Cusick A. Consensus research priorities for cerebral palsy: a Delphi survey of consumers, researchers, and clinicians. Dev Med Child Neurol. 2010;52(3):270–275. 10.1111/j.1469-8749.2009.03358.x. [DOI] [PubMed] [Google Scholar]
- 27.Elwyn G, Crowe S, Fenton M, et al. Identifying and prioritizing uncertainties: patient and clinician engagement in the identification of research questions. J Eval Clin Pract. 2010;16(3):627–631. 10.1111/j.1365-2753.2009.01262.x. [DOI] [PubMed] [Google Scholar]
- 28.Patient Savory C. and public involvement in translative healthcare research. Clin Gov. 2010;15(3):191–199. 10.1108/14777271011063823. [DOI] [Google Scholar]
- 29.Pratt CA, Stevens J, Daniels S. Childhood obesity prevention and treatment: recommendations for future research. Am J Prev Med. 2008;35(3):249–252. 10.1016/j.amepre.2008.05.025. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Claassen CA, Pearson JL, Khodyakov D, et al. Reducing the burden of suicide in the U.S.: the aspirational research goals of the National Action Alliance for Suicide Prevention Research Prioritization Task Force. Am J Prev Med. 2014;47(3):309–314. 10.1016/j.amepre.2014.01.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Schmittdiel JA, Desai J, Schroeder EB, et al. Methods for engaging stakeholders in comparative effectiveness research: a patient-centered approach to improving diabetes care. Healthc (Amst). 2015;3(2):80–88. 10.1016/j.hjdsi.2015.02.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Heiman HJ, Artiga S. Beyond Health Care: The Role of Social Determinants in Promoting Health and Health Equity Menlo Park, CA: The Kaiser Commission on Medicaid and the Uninsured, 2015. http://files.kff.org/attachment/issue-brief-beyond-health-care. Accessed September 30, 2016. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.


