Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2022 Mar 23.
Published in final edited form as: Patient Exp J. 2021;8(1):148–156. doi: 10.35680/2372-0247.1533

Co-production of the quality of patient-centered outcomes research partnerships instrument for people with mental health conditions

Karen L Fortuna 1, Amanda Myers 2, Jessica Brooks 3, Caroline Collins-Pisano 4, Skyla Marceau 5, Sarah Pratt 6, Kathy Lyons 7, Robert Walker 8, Shavon Thompson 9, Kaycie Greene 10, Willie Pringle 11, Katina Carter 12
PMCID: PMC8942114  NIHMSID: NIHMS1734566  PMID: 35330862

Abstract

Mounting scientific evidence over the past decades in the field of psychiatry has shown community engagement in research produces more relevant research, increased uptake of research findings, and better clinical outcomes. Despite the need for the integration of community engagement methodologies into the scientific method, doctoral and master’s level competencies in the field of psychiatry commonly do not include dedicated training or coursework on community engagement methodologies. Without appropriate training or research experience, attempts to facilitate community engagement are often ineffective and burdensome and leave stakeholders feeling disenfranchised. The goal of this study was to co-produce an instrument designed to improve the quality of community engagement research practices by measuring the degree to which researchers have partnered with psychiatric patient stakeholders. The development of the Quality of Patient-Centered Outcomes Research Partnerships Instrument included an iterative co-production process with psychiatric patient stakeholders and scientists, including item formulation, followed by two phases of cognitive interviews with psychiatric patient stakeholders to assess and refine instrument items. A pilot study was conducted to assess acceptability and feasibility. The pilot study of the Patient-Centered Outcomes Research Partnerships Instrument suggested feasibility and acceptability among psychiatric patient stakeholders. The Quality of Patient-Centered Outcomes Research Partnerships Instrument may be a valuable tool to enhance the quality of community engagement research practices within the field of psychiatry.

Keywords: Co-production, patient-centered, mental health conditions, mental health

Introduction

Mounting scientific evidence over the past decades in the field of psychiatry has shown community engagement in research produces more relevant research, increased uptake of research findings, and better clinical outcomes.1–7 Community engagement is defined as “a process of working collaboratively with groups of people who are affiliated by geographic proximity, special interests, or similar situations, with respect to issues affecting their well-being.”1,3 Despite the need for the integration of community engagement methodologies into the scientific inquiry, doctoral and master’s level competencies in the field of psychiatry commonly do not include dedicated training or coursework on community engagement methodologies.8 Without appropriate training or research experience, attempts to facilitate community engagement in research are often ineffective, burdensome, and leave stakeholders feeling disengaged.9

Metrics of community engagement commonly rely on retrospective accounts of stakeholders’ experience working with researchers—including qualitative interviews10–13; process outcomes such as patient stakeholder and faculty training in research, grants funded, and publications,14 and focus groups.5,15–17 Retrospective accounts can only modify community engagement practices after study completion---thus, not offering an opportunity for feedback, critique, and improvement. Further, the current design of retrospective accounts commonly does not take into the account the unique needs of people with mental health conditions,18 including offering reasonable accommodations to offset limited educational backgrounds, low literacy levels, or potential cognitive impairments19 that may impact psychiatric patient stakeholders’ capacity to understand and accurately complete current instruments.

Patient-Centered Outcomes Research Institute (PCORI) is a government-sponsored organization built to focus on and financially support community engagement in research and patient-centered outcomes to help patients make fully informed decisions about their health care. As of 2019, there have been 65 research standards developed to support patient-centered outcomes research, including “Standards Associated with Patient Centeredness” designed to guide researchers in engaging communities and stakeholders to advance patient-centered research.20 Yet, in a survey of researchers (N=103) funded by PCORI, none reported having specific instruments to examine the community engagement process.15

Significant investment in the science of community engagement is needed to examine and improve the community engagement in patient-centered outcomes research (PCOR) process in psychiatric research. Through high-quality partnerships with patient stakeholders, fields of study have developed new insights to address complex issues among patient and stakeholder communities. For example, engaging psychiatric patient stakeholders throughout the research process has the potential to enhance clinical and translational psychiatric research through mutually respectful relationships and shared responsibilities to harness local assets and build healthier communities.21 Yet, not all community engagement approaches are effective or needed with all populations. For instance, methodologies that produce results among highly non-disadvantaged populations (e.g., general population) often result in failure when applied to disadvantaged populations (e.g., American Indians/Native Americans).1,4 Other common research methodologies used to develop programs and services (e.g., focus groups, surveys) that produce positive outcomes among the general population often results in non-relevant research, limited uptake of research findings, and subsequently, poorer outcomes when applied to disadvantaged populations1,4 (e.g., people with mental health conditions). We call this the paradox of the scientific method with vulnerable populations. In general, the more vulnerable and historically marginalized a group, the greater need for patient stakeholder involvement with the decision-making and research activities.33 Thus, when working with people with mental health conditions utilizing a patient stakeholder and researcher partnership framework that shows a high degree of engagement may produce the best program and/or service and clinical outcomes.

Considering the need for and the benefits of community engagement in research,1–7 it is widely recognized that patient inclusion in psychiatric research, as partners, may lead to new advancements in mental health care.22,23 As psychiatric patient stakeholders are increasingly involved in community-engaged research,24 an instrument that takes into account the unique needs of psychiatric patients stakeholders18 and includes opportunities for feedback, critique and improvement is needed to sustain stakeholders’ commitment to PCOR. The goal of this study was to co-produce an instrument through initial item development, cognitive testing, and a pilot study designed to improve the quality of community engagement research practices by measuring the degree to which researchers have partnered with psychiatric patient stakeholders.

Methods

The development of the Quality of Patient-Centered Outcomes Research Partnerships Instrument (QPCOR) included an iterative co-production process based on research methodology for instrument development.25 This co-production process included scientists and patient partners as equal partners with complete decision-making authority in all phases of development and research on QPCOR. Using the Academic-Peer Partnership Model for Community Engagement,26 phase one included a co-production team of four patient stakeholders including people diagnosed with mental health conditions and peer support specialists (i.e., individuals with a mental health condition, trained and accredited by their respective state to offer support services to similar others27), and two scientists skilled in community-engaged research and instrument development discussed the need for the QPCOR. This co-production team delineated and selected methods to design such instrument, and developed the original set of items based on their experiences conducting community-engaged research and principles of community engagement.28

After item formulation, two sets of group cognitive interviews were conducted with psychiatric patient stakeholders over the telephone through a one-hour group cognitive interview guided by the 1st and 5th author. Phase two included five psychiatric patient stakeholders currently engaged in PCOR research projects and phase three included four psychiatric patient stakeholders currently engaged in PCOR research projects. All psychiatric patient stakeholders were emailed the draft QPCOR hours prior to the group cognitive interview and were instructed to read through the instrument prior to attending the one-hour group meeting. In phase four, 16 psychiatric patient stakeholders completed the items electronically and provided feedback. For the pilot in phase four, we administered an online version of the QPCOR to 16 psychiatric patient stakeholders involved in PCOR projects that employ the Academic-Peer Partnership Model for Community Engagement26 (see Figure 1).

Figure 1.

Figure 1.

Open in a new tab

Patient-Centered Outcomes Research Partnerships Instruments Iterative Co-Production Process

The Committee for the Protection of Human Subjects at the [blinded for review] Institutional Review Board approved the project.

Phase One: Item Formulation

Initial item formulation focused on core aspects of the principles of community engagement in research28 and on an analysis of existing community engagement measurement challenges for people with mental health conditions (e.g., usefulness during community engagement process and complexity as determined by terms, number of words used per sentence, number of syllables per word, sentence structure, length). Core elements of community engagement in research included the following item domains (a) purpose, goal, and population; (b) respect/respect community diversity and culture; (c) inclusion/activate community assets; (d) co-learning/develop capacity; (e) become knowledgeable about the community; (f) self-determination; (g) shared-decision making/partner with the community; (h) perceived support/interact and establish relationships with the community; (i) flexibility; and (j) sustainability/commitment to long term collaboration.28 Based on the domains, the co-production team generated several versions of instrument items and explored these elements from patient stakeholders’ perspective through group cognitive interviews.

Phase Two and Three: Group Cognitive Interviews

In an effort to reduce bias, two phases of cognitive interviews with different samples were conducted. The initial items were assessed in phase two. Refined items were assessed in phase three and further modifications made to items. Cognitive interviews are an evidence-based method designed to investigate whether an item accomplishes its planned purpose.29 We wanted to know how individuals would interpret items and assessed their views with regard to whether the item aligned well with principles of community engagement.28

During phase two and phase three group cognitive interviews, participants were given ten minutes to read a set of proposed items before each group call. To elicit information to tailor the instrument to the specific needs and values of the target population, researchers employed verbal probing.30 Verbal probing is an active form of data collection in which interviewers administer a series of questions.29 Standardized, sample verbal probing questions, included “Do the words in the question make sense?”; “Is there anything you find confusing or poorly worded?”; “What does the term ‘healthcare provider’ mean to you”; “What does the term ‘how much effort’ mean to you,” and “In your own words, what do you think the question is asking?”

Phase Four: Pilot Study

In phase four, a final set of items was piloted with a different group of 15 peer support specialists and one patient currently engaged in PCOR research projects to assess acceptability, ease of use, and relevance of items. Of note, this pilot study did not explore and confirm the reliability and validity of this instrument, determine cut-off points, or determine scientifically the ideal timeframe to use QPCOR within PCOR research studies with heterogeneous populations. During phase four, the instrument was emailed to each sample participants’ email address. Participants were given instructions to complete the online instrument.

Data collection and analysis

We recruited a convenience sample of a total of 22 peer support specialists and three people diagnosed with a mental health condition currently engaged as partners in PCOR research projects with the co-production team with the 1st and 5th author. One-hour telephone group cognitive interviews were conducted by two researchers (1st and 5th author), audio-recorded, and transcribed. Participant comments, concerns, misunderstandings and misinterpretations about each item were identified through thematic analysis31 and compared. When no new comments were received in the first interview phase, items were considered finalized, prior to the second interview phase.

Results

Participants

A total of five peer support specialists were interviewed in phase two. Phase three included two people diagnosed with a mental health condition and two peer support specialists currently engaged in PCOR research projects. In phase four, 15 peer support specialists and one person diagnosed with a mental health condition completed the items electronically and provided feedback (see Table 1).

Table 1.

Sample Characteristics of Group Cognitive Interviews and Pilot

1st Group Cognitive Interview (N=5) 2nd Group Cognitive interview (N=4) Pilot (N=16)
Demographic Characteristics
Gender
Male 1 (20%) 2 (50%) 2 (12.5%)
Female 4 (80%) 2 (50%) 14 (87.5%)
Age
18–44 0 (0%) 0 (0%) 4 (0.25)
45–64 5 (100%) 3 (%) 9 (56.25%)
≥65 0 (0%) 1 (%) 2 (12.5%)
Not reported 0 (0%) 0 (0%) 1 (.062%)
Race
African-American 3 (60%) 1 (25%) 0 (0%)
Asian 0 (0%) 0 (0%) 0 (0%)
Caucasian 2 (40%) 3 (75) 12 (75%)
Hispanic 0 (0%) 0 (0%) 0 (0%)
More than one race 0 (0%) 0 (0%) 4 (25%)
Highest Level of Education
Postgraduate 0 (0%) 3 (%) 2 (12.5%)
Bachelor’s 0 (0%) 0 (0%) 6 (37.5%)
Associate’s 0 (0%) 1 (%) 3 (18.75%)
Some college 0 (0%) 0 (0%) 2 (12.5%)
High school diploma 4 (100%) 0 (0%) 3 (18.75%)
Patient Stakeholder Role
Peer support specialist 5 (100%) 2 (50%) 15 (93%)
Person diagnosed with a mental health condition 0 (0%) 2 (50%) 1 (7%)
Open in a new tab

Item Development and Pilot Test

Table 2 below shows how items were initiated, modified and finalized during each phase of QPCOR development (see Appendix for final instrument).

Table 2.

QPCOR Item Development

Domains Original Items Group Cognitive Interview (N=5) Group Cognitive Interview (N=4)
1. Purpose, goal, and population I had a clear understanding of the purpose, goal, and community involved in the research study I had a clear understanding of the purpose of the study. I had a clear understanding of the purpose of the study.
2. Respect I felt listened to I felt listened to I felt listened to
3. Empowerment I received the appropriate training to be an equal partner in the research study. I feel prepared to be an equal partner in the research study. I feel prepared to be an equal partner in the research study.
4. Co-learning/develop capacity Researchers were knowledgeable about people like me. Researchers were knowledgeable about people like me or were willing to learn about people like me. Researchers were knowledgeable about people like me or were willing to learn about people like me.
5. Self-determination I believe that I had choices in how I could engage in the research study. I believe that I had choices in how I could be a part of the research study. I believe that I had choices in how I could be a part of the research study.
6. Shared-decision making/partner with the community I was presented with options of how I could meaningfully help with the research study. I feel prepared to be an equal partner in the research study. I feel prepared to be an equal partner in the research study.
7. Inclusion/activate community assets I felt supported by all members of the research study team. I felt accepted by all members of the research study team. I felt accepted by all members of the research study team.
8. Respect community diversity and culture * Researchers used language that was consistent with my values and culture Researchers used language that was consistent with my values and culture
9. Sustainability/commit to long term collaboration * Both community members and researchers are thinking of ways we can continue to work together in the future Both community members and researchers are thinking of ways we can continue to work together in the future
10. Comfort ** * I felt comfortable engaging with the members of the research study team. I felt comfortable engaging with the members of the research study team.
11. Valued ** * I felt my views were incorporated into the research study I felt my views were incorporated into the research study
Open in a new tab

Note:

*

did not have item at this time point

**

not aligned with community engagement principles.

Approach to Scoring

The researchers originally sought binary yes/no responses for the brief survey; however, phase three sample participants indicated binary responses eliminate the opportunity for nuances in participants’ responses. A scoring mechanism that included a scale of 0–10 could potentially allow for a broad range of responses to questions on the research partnership domains. The co-production team determined a scale of sequential numbers versus “strongly disagree” to “strongly agree” would be less complex to comprehend. To our knowledge, this level of scale development has not been explored scientifically. This scoring method also provides the opportunity to calculate means-based differences between groups.

Pilot

Sixteen participants provided brief demographic details and completed the final version of the Quality of Patient-Centered Outcomes Research Partnerships Instrument (QPCOR). Participants were instruction to consider their PCOR partnerships to date in one specific project and respond on a scale from 0 = “No effort was made by researchers” to 10 = “Every effort was made by researchers.” All participants completed each item in less than one minute. Participants liked that the instrument was quick and potentially gave them a voice in improving the quality of patient-centered research. Participants stated in reference to the QPCOR, “my experiences with patient centered outcome research have been very good. I think these questions are good,” “these questions [sic] will be helpful,” and “it was good they asked the right questions.” Participants recommended items with a score of 6 or lower indicated a need for improvement.

Discussion

The goal of this study was to co-produce an instrument through initial item development, cognitive testing, and a pilot study designed to improve the quality of community engagement research practices by measuring the degree to which researchers have partnered with psychiatric patient stakeholders. Through an iterative co-production process using cognitive interviewing, we co-produced the QPCOR. At the conclusion of the cognitive interviews, we pilot tested the finalized items with a small sample of patient stakeholders and found QPCOR was feasible to implement and acceptable among psychiatric patient stakeholders. This instrument may provide a guide for developing quality PCOR partnerships between psychiatric patient stakeholders and scientists. Examining psychometrics properties of this instrument in future studies may delineate its usefulness in PCOR.

The QPCOR was found to be feasible to implement and acceptable among psychiatric patient stakeholders. QPCOR can be implemented at multiple time points throughout a study to ensure improvements can be made to the partnership during the PCOR. Our co-production team recommends implementation at the beginning of PCOR and then one-month to three-month intervals depending on the length of the study. Items with a score of 6 or lower indicate need for improvement and should be addressed. To date, there are no benchmarks for comparison of results; thus, this manuscript is an important strep, yet a psychometric study of PCOR is an important next phase of development.

Potential benefits of utilizing the QPCOR include real-time account of stakeholders experience working with process—potentially leading to higher-quality, lasting partnerships, novel research questions, new tools and techniques, better clinical outcomes, the establishment of best practices and clinical guidelines, research informed practices for end users, and better uptake of findings26. Examining psychometrics properties of this instrument in future studies may delineate its usefulness in PCOR and lead to potential adaption to other diverse groups (i.e., American Indians/Native Americans or people with hearing, visual, and physical disabilities).

A Likert-type scoring (0–10) used with the QPCOR may be particularly useful in evaluating the differing degrees of community engagement based on the framework of community engagement employed. Community engagement methodology can be viewed as a continuum ranging from low patient stakeholder engagement (e.g., focus groups), medium patient stakeholder engagement (e.g., community engagement studios) to high patient stakeholder engagement2,4 (e.g., community-based participatory research). Participants engaged in research on this continuum may then vary in responses across domains of the QPCOR. Future studies could consider exploring cut-off points based on multiple models of community engagement.

This study is not without limitations. First, while verbal probes are efficient and provide data that is easier to analyze than think-aloud,29 verbal questions may create bias in the subject’s response as verbal probing may lead to more thought-out responses that may not have been collected through survey research methods.29 Verbal probing gives researchers the opportunity to ask questions and expand or follow-up on participants’ answers, which may result in accidentally leading participants towards one answer or another depending on the way in which the probes are presented. Second, bias may also be present due to sample size in instrument development.29 Through an iterative design process commonly accepted, we aimed to reduce bias through a series of interviews and different sample participants. The total sample is consistent with commonly accepted instrument development procedures.34 Third, variation may exist based on participants’ demographics. Exploring the application of the QPCOR with heterogeneous samples may demonstrate variations in utility and outcomes by characteristics (e.g., psychiatric diagnosis, age, peer support specialist or people diagnosed with a mental health condition status, educational level, cognitive status). Last, because of the small sample size of participants enrolled in the pilot phase, we cannot yet establish the psychometric validity of this instrument. Future research should recruit a large, diverse sample to evaluate and confirm the reliability and validity of this instrument, determine cut-off points, and determine scientifically the ideal timeframe to use QPCOR within PCOR research studies with heterogeneous populations.

The QPCOR was co-produced to assess psychiatric patient stakeholder involvement in research partnerships. This is the first ever study on developing an instrument for patient stakeholder partnerships in psychiatric research. This study followed the guidelines it set forth by promoting equal partnerships throughout the research process. From co-producing the original items for the community engagement measure, the two group cognitive interviews, and the pilot study, the authors and patient stakeholders actively engaged in the model that is set forth. The instrument presented may provide a general guide for community-engaged research with psychiatric patient stakeholders and may supplement training that academic professionals currently lack while not placing undue burden on patient stakeholders involved in the process.

Support

Dr. Karen Fortuna is supported by the following funding: K01MH117496 & PCORI/PCS-2017C2-7724-IC

Also, Karen Fortuna Skyla Marceau, Sarah Pratt, Kathy Lyons, Robert Walker, Shavon Thompson, Kaycie Greene, Willie Pringle, Katina Carter were supported by PCORI/PCS-2017C2-7724-IC

Cover Page Footnote

Dr. Karen Fortuna is supported by the following funding: K01MH117496 & PCORI/PCS-2017C2–7724-IC Also, Karen Fortuna Skyla Marceau, Sarah Pratt, Kathy Lyons, Robert Walker, Shavon Thompson, Kaycie Greene, Willie Pringle, Katina Carter were supported by PCORI/PCS-2017C2-7724-IC This article is associated with the Innovation & Technology lens of The Beryl Institute Experience Framework (https://www.theberylinstitute.org/ExperienceFramework). You can access other resources related to this lens including additional PXJ articles here: http://bit.ly/PX_InnovTech

Appendix

Quality of Patient-Centered Outcomes Research Partnerships Instrument Instructions:

Consider your Patient-Centered Outcomes Research partnerships to date in one specific project and respond to each of the following questions on a scale from 0 = “No effort was made by researchers” to 10 = “Every effort was made by researchers”. You do not need to place your name on the instrument. Of note, implement this instrument at multiple time points throughout your study, including the beginning of PCOR and at one-month or three-month intervals depending on the length of the PCOR.

Question Write in a Score 0–10

  1. I had a clear understanding of the purpose of the study. ______

  2. I felt listened to ______

  3. I feel prepared to be an equal partner in the research study. ______

  4. Researchers were knowledgeable about people like me or were willing to learn about people like me. ______

  5. I believe that I had choices in how I could be a part of the research study. ______

  6. I feel prepared to be an equal partner in the research study. ______

  7. I felt accepted by all members of the research study team. ______

  8. Researchers used language that was consistent with my values and culture ______

  9. Both community members and researchers are thinking of ways we can continue to work together in the future ______

  10. I felt comfortable engaging with the members of the research study team. ______

  11. I felt my views were incorporated into the research study ______

Scoring: Items with a score of 6 or lower should be addressed PCOR teams

Contributor Information

Karen L. Fortuna, Dartmouth College

Amanda Myers, Brandeis University.

Jessica Brooks, University of Wisconsin.

Caroline Collins-Pisano, Dartmouth College.

Skyla Marceau, Dartmouth College.

Sarah Pratt, Dartmouth College.

Kathy Lyons, Dartmouth College.

Robert Walker, MA Department of Mental Health.

Shavon Thompson, Centerstone, Kentucky.

Kaycie Greene, Centerstone, Tennessee.

Willie Pringle, Centerstone, Tennessee.

Katina Carter, Centerstone, Kentucky.

References

  • 1.Cyril S, Smith BJ, Possamai-Inesedy A, Renzaho AM. Exploring the role of community engagement in improving the health of disadvantaged populations: a systematic review. Global health action. 2015;8:29842. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Woolf SH, Purnell JQ, Simon SM, et al. Translating evidence into population health improvement: strategies and barriers. Annu Rev Public Health. 2015;36:463–482. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.MacQueen KM, Bhan A, Frohlich J, Holzer J, Sugarman J, Ethics Working Group of the HIVPTN. Evaluating community engagement in global health research: the need for metrics. BMC Med Ethics. 2015;16:44. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Khodyakov D, Mikesell L, Schraiber R, Booth M, Bromley E. On using ethical principles of community-engaged research in translational science. Translational research : the journal of laboratory and clinical medicine. 2016;171:52–62.e51. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Salihu H, Salinas-Miranda A, Wang W, Turner D, Berry E, Zoorob R. Community Priority Index: utility, applicability andvalidation for priority setting in community-based participatoryresearch. J Public Health Res. 2015;4(2):443. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Holzer J, Ellis L, Merritt M. Why we need community engagement in medical research. J Investig Med. 2014;62(6):851–855. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Kwon S, Tandon S, Islam N, Riley L, Trinh-Shevrin C. Applying a community-based participatory research framework to patient and family engagement in the development of patient-centered outcomes research and practice. Translational behavioral medicine. 2018;8(5):683–691. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Ahmed S, Palermo A. Community engagement in research: frameworks for education and peer review. American journal of public health. 2010;100(8):1380–1387. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Staley K Exploring Impact: Public Involvement in NHS,. Public Health and Social Care Research. 2009; Eastleigh, UK INVOLVE. [Google Scholar]
  • 10.Sanchez V, Carrillo C, Wallerstein N. From the ground up: Building a participatory evaluation model. Progress in Community Health Partnerships: Research, Education, and Action. 2011;5(1):45–52. [DOI] [PubMed] [Google Scholar]
  • 11.Goodman R, Speers M, Mcleroy K, et al. Identifying and defining the dimensions of community capacity to provide a basis for measurement. . Health Education & Behavior. 1998;25(3):258–278. [DOI] [PubMed] [Google Scholar]
  • 12.Khodyakov D, Stockdale S, Jones A, Mango J, Jones F, Lizaola E. On measuring community participation in research. Health Educ Behav. 2013;40(3):346–354. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.McCloskey D, McDonald M, Cook J, et al. Community engagement: Definitions and organizing concepts from the literature The Centers for Disease Control and Prevention; 2012. [Google Scholar]
  • 14.Eder M, Evans E, Funes M, et al. Defining and Measuring Community Engagement and Community-Engaged Research: Clinical and Translational Science Institutional Practices. Progress in community health partnerships : research, education, and action. 2018;12(2):145–156. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Sprague Martinez L, Carolan K, O’Donnell A, Diaz Y, Freeman ER. Community engagement in patient-centered outcomes research: Benefits, barriers, and measurement. Journal of clinical and translational science. 2018;2(6):371–376. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Esmail L, Moore E, Rein A. Evaluating patient and stakeholder engagement in research: moving from theory to practice. Journal of comparative effectiveness research. 2015;4(2):133–145. [DOI] [PubMed] [Google Scholar]
  • 17.Sandoval J, Lucero J, Oetzel J, et al. Process and outcomeconstructs for evaluating community-based participatory re-search projects: a matrix of existing measures. Health Educ Res. 2012;27(4):680–690. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Rotondi A, Spring M, Hanusa B, Eack S, Haas D. esigning eHealth Applications to Reduce Cognitive Effort for Persons with Severe Mental Illness: Page Complexity, Navigation Simplicity, and Comprehensibility. JMIR human factors. 2017;4(1). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Galderisi S, Davidson M, Kahn R, et al. Correlates of cognitive impairment in first episodeschizophrenia: The EUFEST Study. Schizophr Res. 2009;115(2–3):104–114. [DOI] [PubMed] [Google Scholar]
  • 20.Wu A, Bingham C. PCORI Methodology Standards: Academic Curriculum. 2016; https://www.pcori.org/sites/default/files/PCORI-Methodology-Standards-Curriculum-PatientCenteredness-3.pdf.
  • 21.Horowitz C, Robinson M, Seifer S. Community-based participatory research from the margin to the mainstream: are researchers prepared? Circulation. 2009;119(19):2633–2642. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Dubois J, Bailey-Burch B, Bustillos D, et al. Ethical issues in mental health research: the case for community engagement. Curr Opin Psychiatry. 2011;24(3):208–214. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Lincoln A, Wallace L, Kaminski M, Lindeman K, Aulier L, Delman J. Developing a community-based participatory research approach to understanding of the repeat use of psychiatric emergency services. Community Ment Health J. 2016;52(8):1015–1021. [DOI] [PubMed] [Google Scholar]
  • 24.Fortuna K, Barr P, Goldstein C, et al. Application of Community-Engaged Research to Inform the Development and Implementation of a Peer-Delivered Mobile Health Intervention for Adults With Serious Mental Illness. J Participat Med. 2019;11(1):e12380. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Elwyn G, Barr P, Grande S, Thompson R, Walsh T, Ozanne E. Developing CollaboRATE: a fast and frugal patient-reported measure of shared decision making in clinical encounters. Patient Educ Couns. 2013;93(1):102–107. [DOI] [PubMed] [Google Scholar]
  • 26.Fortuna K, Barr P, Goldstein C, et al. Application of Community-Engaged Research to Inform the Development and Implementation of a Peer-Delivered Mobile Health Intervention for Adults With Serious Mental Illness. J Participat Med 2019;11(1):e12380. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Solomon P Peer support/peer provided services underlying processes, benefits, and critical ingredients. Psychiatr Rehabil J. 2004;27(4):392–401. [DOI] [PubMed] [Google Scholar]
  • 28.Control CfD. Principles of Community Engagement. 2011; https://www.atsdr.cdc.gov/communityengagement/pdf/PCE_Report_508_FINAL.pdf. Accessed May 20, 2020.
  • 29.Willis G, Artino A. What Do Our Respondents Think We’re Asking? Using Cognitive Interviewing to Improve Medical Education Surveys. Journal of graduate medical education. 2013;5(3):353–356. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Priede C, Farrall S. Comparing results from different styles of cognitive interviewing: ‘verbal probing’ vs. ‘thinking aloud.’ International Journal of Social Research Methodology. 2011;14(4):271–287. [Google Scholar]
  • 31.Vaismoradi M, Turunen H, Bondas T. Content analysis and thematic analysis: Implications for conducting a qualitative descriptive study. Nurs Health Sci. 2013;15(3):398–405. [DOI] [PubMed] [Google Scholar]
  • 32.Davidson L, Bellamy C, Guy K, Miller R. Peer support among persons with severe mental illnesses: a review of evidence and experience. World Psychiatry. 2012;11(123–128). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33.Flicker S, Travers R, Guta A, McDonald S, Meagher A. Ethical dilemmas in community-based participatory research: recommendations for institutional review boards. Journal of urban health : bulletin of the New York Academy of Medicine. 2007;84(4):478–493. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Davis A Instrument development: getting started. J Neurosci Nurs. 1996;28(3):204–207. [DOI] [PubMed] [Google Scholar]

RESOURCES