Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2021 Dec 16.
Published in final edited form as: BMJ Qual Saf. 2019 Dec 11;29(9):746–755. doi: 10.1136/bmjqs-2019-010059

Logic model framework for considering the inputs, processes and outcomes of a healthcare organisation-research partnership

Amir Alishahi Tabriz 1, Susan A Flocke 2,3, Deirdre Shires 4, Karen E Dyer 5, Michelle Schreiber 6,7, Jennifer Elston Lafata 8
PMCID: PMC8675565  NIHMSID: NIHMS1752321  PMID: 31826921

Abstract

Background.

The published literature provides few insights regarding how to implement or consider the effects of knowledge co-production partnerships in the context of delivery system science.

Objective.

To describe how a health care organization—university-based research partnership was developed and used to design, develop and implement a practice-integrated decision support tool for patients with a physician recommendation for colorectal cancer screening.

Design:

Instrumental case study.

Participants:

Data were ascertained from project documentation records and semi-structured questionnaires sent to 16 health care organization leaders and staff, research investigators, and research staff members.

Key Results.

Using a logic model framework, we organized the key inputs, processes and outcomes of a health care organization—university-based research partnership. In addition to pragmatic researchers, partnership inputs included a health care organization with a supportive practice environment and an executive-level project sponsor, a mid-level manager to serve as the organizational champion, and continual access to organizational employees with relevant technical, policy and system/process knowledge. During program design and implementation, partnership processes included using project team meetings, standing organizational meetings, and one-on-one consultancies to provide platforms for shared learning and problem solving. Decision-making responsibility was shared between the health care organization and research team. We discuss the short-term outcomes of the partnership, including how the partnership affected the current research team’s knowledge and health system initiatives.

Conclusion.

Using a logic model framework, we have described how a health care organization—university-based research team partnership was implemented. Others interested in developing, implementing, and evaluating knowledge co-production partnerships in the context of delivery system science projects can use the experiences to consider ways to develop, implement and evaluate similar co-production partnerships.

Keywords: Delivery system science, health care stakeholder engagement, implementation science, knowledge co-production, quality improvement

INTRODUCTION

Traditional sequential, linear models of research-to-practice result in long delays and gaps in the translation of research-generated knowledge.(1,2) Voids in the use of shared decision-making and decision aids represent such a gap: Despite strong ethical reasons and evidence of their benefits,(3,4) both are rarely used in practice.(5–7) Barriers to implementation include clinician knowledge gaps and time constraints (5,6), and factors that limit the practicality of their use, including challenges integrating decision support within clinic workflows(8,9). How to better support decision aid implementation remains an ongoing challenge.

Participatory research (10) and knowledge co-production (11) models are advocated as alternative models to facilitate research translation.(12–16) Within knowledge co-production models, both researchers and practitioners are considered experts, each bringing their complementary knowledge, skills, insights and professional networks to the team. In the context of health care delivery research, if interventions are designed without involving those who deliver care, this means interventions are developed without input from those who have an understanding of the complex context of care delivery. In a knowledge co-production model, this contextual knowledge is considered in hopes of increasing relevance and usability. In such partnerships, practitioners and researchers work together to solve problem-driven issues with the expectation that the research will generate solutions that easily translate into practice or policy. (17–22)

Although conducting research within a partnership has become popular,(23,24) the literature is void of insights regarding how such health care organization - research partnerships can be operationalized.(25,26) This is particularly true for health service researchers wanting to partner with care delivery organizations.(27) The result is a general gap in knowledge regarding the implementation of knowledge co-production strategies.(22) To address this gap, we used a logic model framework to illustrate how we operationalized a health care organization - research partnership during the design, development and implementation phases of an NCI-funded project (R01CA197205), the e-assist: Colon Health study.

METHODS

The e-assist: Colon Health study has been described in detail elsewhere.(28) In brief, we developed a post-office visit, colorectal cancer (CRC) decision support program for patients with a physician recommendation for CRC screening. The program was embedded within the health care organization’s electronic health record (EHR) with patients accessing it via an online patient portal. Program content reinforced CRC screening benefits, compared the benefits and risks of screening alternatives, and addressed the common questions and barriers patients have after receiving a physician recommendation for CRC screening (29). Program content included short videos addressing reasons for screening and common barriers to screening. Much of the program was self-directed, enabling users to view content in their preferred order and supplemental material (e.g., tips for how to complete screening) only if it was of interest.

We used an instrumental case study approach (30) to describe how the partnership was developed and implemented. The instrumental case study is an established research approach in which the study of a specific case is used to produce a broader understanding of a complex phenomenon (31). Using this approach enabled us to provide an in-depth understanding of how health care organization-research partnerships can be operationalized (30,31).

To obtain information for the case study, we reviewed project-meeting minutes and routine decision notes/logs maintained by the research staff, and administered a semi-structured questionnaire to project team members (Appendix A). We used the questionnaire to solicit information regarding individuals’ goals and priorities for the development of the program, and their perceptions of the processes used during the partnership (e.g., we solicited descriptions of the working relationships and how health care employees were involved). We also inquired about challenges faced, and how project participation may have enhanced knowledge and skills. Furthermore, because a specific goal of working in partnership with the health care organization was to ensure the decision support was integrated with clinic workflows, we asked respondents how they perceived the e-assist program fit (or did not fit) with clinic workflows and processes. A post-doctoral fellow (AAT) who was not involved with the project during the design, development or implementation phases emailed the questionnaire to health care organization leaders and staff who routinely partnered with the research team (n=4), each of the research investigators on the team (n=7), and key research staff members (n=5). We used a thematic analysis approach (32) to analyze questionnaire responses. We developed a preliminary codebook of emergent topics (33), coded important passages into related theme(s), and selected illustrative quotations when they were explanatory of each theme. Finally, we used a logic model approach (34) to organize the key components of the partnership, and to structure a framework that depicted key inputs to, processes of, and outcomes from the partnership (Figure 1).

Figure 1.

Figure 1.

Open in a new tab

Delivery System Sciences Framework

Activities reported here were part of a planned formative evaluation approved by the health system’s Institutional Review Board. An introductory email sent to questionnaire recipients explained that information was being collected to understand how the program partnership was developed, and that responses would be used to describe how the partnership was used to develop and test the e-assist program. Twelve (75%) of those sent the questionnaire (3 health care organization employees, 4 research investigators, and 5 research staff) provided responses.

RESULTS

Inputs

Research-ready and Supportive Organization

During project conceptualization, the university-based research team sought to identify a ‘research-ready’ and supportive health care organization partner. We perceived a ‘research ready’ organization as a manifestation of a learning health care system (35), defined as an organization within which “science, informatics, incentives, and culture are aligned for continuous improvement and innovation.” (36) As such, a ‘research ready’ organization was one that understood the importance of research and wanted not only to use the results of research, but also actively participate in how the research was designed, conducted, and made available to those delivering care, In addition to identifying an organization with a supportive improvement and innovation culture, it was also important to identify an organization that could embrace and provide an environment that was supportive of the organizational change needed to support the specific research topic and questions (37). The first entry in Table 1 provides an illustrative quotation regarding the research-readiness of the partnering health care organization (Table 1, IQ 1).

Table 1.

Illustrative Quotes from Key Informant Questionnaire Responses

Domain Quotation Number Illustrative Quotation (Participant’s role, Study Participant Number)
Research-ready and Supportive Organization 1
“Because the [name of the health system] has such a research-friendly culture, I found that the nurse managers, clinicians, and administrators I worked with were helpful in problem-solving and directing me to the appropriate health system employees as well as interested in and invested in the study. They all had ideas about how the study could complement care already being provided and how to best integrate the study into current practice.”
(Research staff 2)
Pragmatic and Responsive Researchers 2
“In return, when asked, members of the research team participated in the Health System’s Quality Expo and provided routine updates that were of relevance to the Health System’s operations.”
(Research investigator 2)
3
“… [the health care organization team] had a close working relationship with the research team. More than most research projects, we were co-developers of the initiative and partners throughout the process, from conceptualization, to development of the tool, to testing and evaluation. It was critical for the research team to be able to understand our workflows in the design and to understand our patient’s perspectives.”
(Health care organization staff 1)
Core Health Care Organization Team Members 4
“The partnership with the Chief Quality Officer [CQO] was critical to our ability to integrate the decision support within existing workflows. She was able to help us understand organizational priorities and interests. By aligning how we built with program with the organization’s own development and quality goals, we were able to work in a mutually beneficial way.”
(Research investigator 4)
Shared Responsibilities and Commitment 5
“We are interested in the programmer working with us and they saw value in doing so in terms of subsequent applications. They wanted to learn how to use their shiny new toy and we wanted to make what we were doing fit with practice.”
(Research investigator 4)
Communication and Decision Making 6
“Constant communication was key in getting to the final product and has remained important as we continue to implement changes.”
(Research staff 3)
7
“I felt that the research team was diligent in seeking out feedback and collaborative opportunities with as many stakeholder groups as possible, being responsive to their clinical needs and challenges, and approaching the entire process as a real partnership with the health care delivery team. This partnership was characterized by frequent communication, feedback-seeking, consensus-building and joint decision-making.”
(Research staff 1)
Shared Learning and Problem-Solving Opportunities 8
“We were connected with key staff members who willingly provided additional insights into local context, resources and capabilities. The Nurse Manager helped us navigate how best to get input and support from existing primary care leaders and to use an existing quality committee to ensure that program content was consistent with local practices.”
(Research investigator 4)
9
“The really nice thing about e-assist is that it’s an automated program. It automatically goes out to the patient when the provider places an order for an open access referral or stool test. There’s nothing additional the provider needs to do to enable e-assist for a patient E-assist is interactive, and provides the patient with the opportunity to learn as much (or as little) as they desire about colon cancer screening.”
(Health care organization staffs)
Outcomes 10
“I learned more about what is and isn’t possible with [EHR] questionnaire functionality, including how to embed videos! I learned how to automatically trigger [patient portal] messages with questionnaires attached. I thought the use of images in place of prompts was a very creative solution, and something that I hadn’t thought of before.”
(Health care organization staff 3)
11
“The participation has enhanced our team’s knowledge of electronic capabilities of shared decision making tools and opened up a potential new approach to assisting patients. We also learned much more about the complexities of the research process, and at the same time had to examine in detail our own workflows in order to best design the tool. Over time we aim to learn about our patient needs and preferences through the use of these tools, and hope to expand this to other clinical questions in the future.”
(Health care organization staff 1)
12
“Participation gave me new words to describe working together with health systems to solve quality improvement issues - e.g. practice-embedded research. It also led to discussions to better express the processes of engagement and collaboration across stakeholders that helped to propel other projects forward.”
(Research investigator 2)
Open in a new tab

Because organizational context has been found critical to the success (or failure) of projects that rely on co-production strategies (38–40), it was crucial that members of the research team be familiar with the health care partner’s organizational culture and priorities. Such familiarization can often best come from an established relationship.(40) In this case, the principal investigator’s (PI’s) knowledge of the partnering health system, and existing relationships with members of their leadership team, led her to reach out to the system’s Chief Quality Officer (CQO) during project development.

Conversations between the PI and the CQO confirmed an organizational interest in co-developing new ways to support patient decision making, if that support could enable the health care organization to leverage the capabilities of a newly purchased EHR. These conversations not only helped shape the direction of the grant application but established clear expectations and boundaries for the project. Importantly, they as served as the foundation for a trusting and collaborative relationship.

Pragmatic and Responsive Researchers

In addition to identifying a ‘research ready’ organization, it was important that the research team be pragmatic in that they had realistic expectations of what could be achieved, and how the health care organization staff could assist. We achieved this, by ensuring that the research team collectively not only had the academic training and experiences needed to design and test the decision support program, but also understand the organization’s operational needs and priorities. As such, the research team included practicing clinicians and those with experience working as embedded health services researchers. Members of the research team also had experience using quality improvement approaches (e.g., process mapping) and anthropological field methods (e.g., observation) important to garnering an understanding of organizational context and lived experiences.

Being pragmatic also required an understanding of organizational priorities and being respectful of those priorities and health care organization staff time. For example, while the research team was committed to grounding the program in behavioral theory and it being practice integrated, they were flexible regarding how practice integration was achieved. Because of the health care organization’s priorities and interests, prior to grant submission it was decided that the proposed decision support program would leverage the platform of the EHR-embedded, online patient portal. This decision enhanced the potential for practice integration while affording an opportunity for health care organization employees to develop their own ability to leverage their EHR environment for other innovations. The result was an intervention grounded conceptually within behavioral theory yet feasible to integrate within the delivery system. In addition, during program development, the research team routinely provided updates to organizational staff, and when asked by organizational representatives to submit abstracts to and present at internal learning and quality forums, did so. As such, the project was designed to be meaningful to the researchers and health care organization (Table 1, IQ 2 and 3).

Core Health Care Organization Team Members

Senior executive-level support was crucial to the project’s success as such an individual could articulate how study participation benefited the organization, identify an internal project champion, protect the time of key individuals, and serve as a liaison with others in the organization. Such functions were crucial given the desire for active participation from those with direct knowledge of the relevant care system and processes, as this was deemed necessary for practice integration.

With the support of the CQO, the project had the equivalent of a quality improvement (QI) sponsor within the organization.(41) Her participation enabled the research team to understand organizational challenges, and the strategic directions of the organization (Table 1, IQ4). Upon the recommendation of the CQO, the research team also partnered with a nurse manager who worked within the health care organization’s Information Technology division. Establishing the latter relationship prior to securing funding ensured engagement of a person who oversaw frontline staff and who was supervised by a member of the senior management team (42). As such, the nurse manager could serve as a project champion with local knowledge of QI initiatives and processes within the health care organization. Her role knowledge of the organization was instrumental to the project, as was her ability to garner support for the project among and within key quality-oriented leaders and committees.

Once funding was secured, an EHR programmer, who reported to the nurse manager, received partial salary coverage from the grant. This ensured regular health care staff representation during project meetings, solidified organizational commitment to project success, and enhanced information exchange between the health care organization and research team. Including these three individuals within the core project team enabled continual insights into the organization’s priorities, challenges, and strategic directions—each important consideration for multi-year, practice-embedded delivery system science studies.

Engaging Knowledgeable Stakeholders and Content Experts

While core team members were instrumental inputs to the co-production process, they alone could not provide the organizational content expertise and knowledge needed. Thus, core team members’ organizational networks and connections were critical to the project team’s ability to solicit input from clinicians and other content experts. Through these connections, the project team was introduced to individuals with relevant technical, clinical and administrative expertise. Those individuals were involved in consultancy roles, having no decision-making authority per se, but instead influencing decisions made by the project team.(43) We used this type of involvement, for example, to understand the feasibility of information exchange across the Health System’s firewall as the project team explored where to house video content contained within the decision support program.

Processes

Shared Responsibilities and Commitment

The team’s commitment to developing a program that could be sustained by the health care organization post research funding served as a backdrop for all subsequent decisions. For example, as indicated above, a joint decision was made to build the e-assist: Colon Health program within the health care organization’s EHR. This decision was important to the CQO as it meant EHR infrastructure could be used to integrate program and clinic workflows without additional resource allocation. Furthermore, this enabled the health care organization to advance their EHR and patient portal capabilities—something desired by senior leadership. As such, an interdependency existed between the research team and health care organization, as each one needed each other to move forward their individual agendas (Table 1, IQ 5). Despite this interdependency, the reliance on the health system’s EHR resulted in an imbalance of power. Nonetheless, because of the pragmatism of the research team and the shared responsibilities and commitment to the project, when challenges arose (e.g., during scheduled EHR upgrades or other organizational competing demands) we were able to work together to modify timetables and programs to accommodate competing priorities while still achieving research milestones.

Communication and Decision Making

Communications within partnerships are characterized fundamentally by being respectful, open and honest, but also by their bi-directional nature.(44) The latter is characterized importantly by all team members initiating communication exchanges, not just the research team. For example, within the e-assist partnership, shortly before the trial was scheduled to begin pilot testing, the research team received an email from the nurse manager alerting them that the CRC screening policies within the health care organization were likely to be changed (Figure 2). This enabled the research team to adapt program content, keeping trial materials relevant and consistent with clinic practices (Table 1, IQs 6 and 7). Similarly, open and honest communications ensured that results from patient focus groups that were used to inform program design, were shared with health care organization staff as they became available. Among other things (45), these results highlighted patients’ preferences for a program logo that “looked like” the health system’s logo. However, because some health system staff raised concerns regarding trademark infringement, the research team redirected its logo development efforts. Doing so enabled us to avoid a potentially lengthy project delay while organizational approval was sought and/or compromising the established working relationship.

Figure 2.

Figure 2.

Open in a new tab

Email showed communications within a partnership

Shared Learning and Problem-Solving Opportunities

A key component of the partnership was the development of opportunities for relationship building and shared learning.(46) These opportunities facilitated joint problem solving, and shared and transparent decision making. Having such opportunities required planning time for interactions among core project team members, and between research team members and health care organization leaders and staff. Core health care organization team members identified relevant standing committees and other forums that could be used as sounding boards, and core research team members made time to be present, listen and learn. For example, the Health System’s Ambulatory Care and Gastroenterology Quality Committees were leveraged both as sounding boards and for organizational endorsement and approvals. As entities with decision-making authority, these committees were approached during their regular meetings to secure feedback and approval on intervention design, content, and implementation tactics. Because the former committee meets via teleconference, off-site research team members were able to Join meetings remotely and for the full meeting duration—enabling research team members to better understand organizational priorities and context (Table 1, IQ 7). Multiple decisions regarding program-related workflow and content were discussed as agenda items within these committee meetings, enabling research team members and organizational staff jointly to solve challenges and make decisions. Members of these committees, as representative ‘end users,’ were also approached individually for input and feedback.

Because of the mutual commitment to project success, all members of the project team engaged in a dynamic and iterative information-sharing and learning process. The research team worked iteratively with the nurse manager and her local network to map the CRC screening process (Table 1, IQ8). Because of this mapping, the project team identified and sought input from additional people whose work might be affected by the patient-targeted decision support program. Similarly, the research team’s need to randomly allocate eligible patients across study arms for the evaluation led to discussions among relevant research and organizational content experts. While the need arose because of a research need, in jointly working towards a solution, the health care organization gained valuable knowledge and experience relevant to targeting programming content in general.

The use of process mapping and the open and ongoing bi-directional communications enabled program content to be consistent with the organization’s current screening practices. It also enabled program content to ascertain where within the screening process, and thus clinic workflows, a patient was at the time of program engagement. Knowing this enabled program content to provide appropriate organizational contact information should patients have questions regarding test completion or appointment scheduling, further enhancing program integration. Similarly, discussions with members of the health care organization’s Primary Care and Gastroenterology Quality Committees—combined with the team’s understanding of the colonoscopy referral process—were used to determine which patients with a physician order for CRC screening it would be most appropriate for the program to target. The result has been a decision support program that fits with existing clinic workflows and processes (Table 1, IQ 9).

Outcomes

When considering the short-term effects and long-term impact of a partnership, we considered benefits that accrue to patients, research team members, health care organizational staff, and the organizations more broadly (Figure 1). For example, one health care staff member described how working with the research team increased his/her knowledge about portal capabilities (Table 1, IQ 10). We have also observed that as a result of the partnership, both the health care organization and research team increased their capacity for innovation (Table 1, IQ 11). (47) Similarly the partnership has also provided opportunities to conduct research that was of relevance to the health care organization.(22) For example, not only has the partnership led to two joint publications,(28,48) it also led to a newly launched organizational initiative focused on eliminating racial disparities in portal access, and subsequent grant applications. The partnership has also left team members better prepared for subsequent partnerships (Table 1, IQ 12).

Although it is too soon to assess within thee-assist project, the long-term goal of such a partnership is the rapid translation of relevant research findings into practice and ultimately, improved health. Our experiences to date, have pointed to opportunities for additional organizational and research team impact, especially in terms of knowledge/skill development and its spillover into future projects.

DISCUSSION

As acknowledged in the patient and family engagement literature (43), engagement and partnership can happen across a continuum characterized by how much information flows, how active a role individuals play in decision making, and how involved the individuals become in the broader context. In this paper, we used a logic model framework to describe the inputs, processes and outcomes associated with a health care organization—university-based research team partnership. As such, the case study reported here provides a novel in-depth description of the design and implementation of a delivery system science partnership that was characterized by bi-directional information flow, shared decision-making authority, opportunities for shared problem solving, and mutual project responsibility.

Important to the ability to develop the partnership was the compatibility of the project with the organization’s priorities and values. As others have proposed,(49,50) during the design and development phases of the project, the research team considered the compatibility of their ideas with different healthcare organizations’ norms, values, and needs, to make sure the project was likely to fit culturally and logistically within the organization. Selecting a compatible learning healthcare system (i.e., a research-ready organization) made it feasible to adapt, tailor and refine the project to be consistent with organizational needs and priorities.(20,40,51) Learning healthcare systems by definition have the structure to support communications among patients, providers and researchers, and the ability to leverage their EHR to conduct practice-embed research (52) both of which were important to the current research efforts. It is, however, important to note, that an organization that is research-ready in the context of one research project may not be ready to support a research partnership for a different project. This is the case, because in addition to having a culture and infrastructure that is compatible with a learning health system, the organization also has to have an interest in the success of the specific research project. This vested interest in the specific project can help minimize conflicts and problems during the project. Thus, as others have recommended, (18) we found the use of a scoping phase for familiarization and relationship building important to partnership success. Being pragmatic during this scoping phase also enabled us to facilitate the commitment of the organization to the research. In this case, multiple in-person, phone and email discussions among the PI, research project manager, nurse manager, and CQO took place during grant development. These communications ensured a shared understanding of research team and health care organization priorities, and enabled the team to develop a shared vision for the patient-centered CRC screening decision-making support program.(40)

Such health care organization-research partnerships likely are useful in situations where challenges with practice translation have been well documented(53). They also likely are useful when a user-centered design process could enhance program applicability and adoptability, and thus sustainability. By explicitly incorporating the needs and desires of the health care organization during the design, development, and implementation phases of the project, we were able to develop and implement a theory-based, decision support program that is integrated with clinic processes and workflows. As such, we were able to address previously identified barriers to implementing decision aids in practice, including physician concerns regarding time constraints and practicality (5,6,8,9). To do so, required compromises and an understanding that research rarely per se can be a priority for health care organization and their employees. The research team’s understanding of this, and that power dynamics are often a factor within the practice of knowledge co-production (54,55), led us to be sensitive to the organization’s needs and priorities and to continually seek to avoid confrontation. Furthermore, because power imbalance can jeopardize team outcomes (56,57) when challenges arose, we worked to find solutions that minimized operational disruptions and for which organizational partners indicated support.

In addition, following the recommendations of previously successful projects,(58–61) we created a culture of partnership and collaboration with key health care organization leaders during the project’s initial phases. Having an executive-level person along with a mid-level manager, as a sponsor and champion, respectively, helped to provide the organization with a sense of project ownership and ensured important project advocates at multiple levels of the organization. Such middle managers, in particular, can use their unique organizational positions to promote the adoption and implementation of the project, serve as a bridge between the health care organization and research team, help to engage different stakeholders, and mediate day-to-day activities. Others establishing such partnerships should not overlook their importance.

Limitations

Within the partnership, we tailored with whom and how we engaged to the organization’s structure and culture as well as to the specific research project. How these may need to be altered for other organizational contexts and projects cannot be determined from an instrumental case study. The in-depth description of program input considerations and partnership processes, however, are rarely discussed in the literature and our experiences may provide an important foundation upon which future partnership processes can be built and assessed. The partnership and experiences, nonetheless, remain a case study and care should be taken when generalizing specific aspects of the approach to other delivery system science research projects.

CONCLUSION

The e-assist: Colon Health study provided a unique opportunity to study how a health care organization—research partnership could be developed, implemented, and ultimately used to enhance the adoptability and implementation of an intervention in the context of health care delivery science studies. As illustrated by our experiences, a logic model framework can be useful for identifying the inputs needed for a successful partnership, what key processes are important to operationalize the partnership, and the outcomes that can be considered when determining the impact of a partnership. Others interested in developing, implementing, and evaluating knowledge co-production partnerships in the context of delivery system science projects can adopt the logic model framework presented here to consider ways to develop, Implement and evaluate similar co-production partnerships.

Appendix A. Open-ended Key Informant Questionnaire Items for Formative Evaluation

  1. Please describe your working relationship with members of the research team or health care delivery team?

  2. What were your goals and priorities for the development of the e-assist program?
    • How, if at all, did these change over time?
  3. How does the e-assist program fit (or not fit) with clinic workflows and processes?
    • Please think about any aspect of the program, its content, who the program targets, its timing, its use of the EHR/patient portal, etc.

  4. What challenges did the project face and how were they overcome?

  5. How has your participation with the project enhanced your or your teams’ knowledge and skills?
    • What did you learn about EHR/portal capabilities?
    • What did you learn about clinic processes?
    • What did you learn about research methods?
    • What did you learn about the patient’s needs or preferences?
  6. How were health system leaders and staff involved with the project?”
    • Would you provide an example of how they were involved and what their roles were?
    • How did their involvement help or hinder the project?

References

  • 1.Grol R, Grimshaw J. From best evidence to best practice: effective implementation of change in patients’ care. The lancet. 2003;362(9391):1225–30. [DOI] [PubMed] [Google Scholar]
  • 2.Grol R. Successes and failures In the implementation of evidence-based guidelines for clinical practice. Med Care. 2001;39(8):II-46–II-54. [DOI] [PubMed] [Google Scholar]
  • 3.Shay LA, Lafata JE. Where is the evidence? A systematic review of shared decision making and patient outcomes. Med Decis Making. 2015;35(1):114–31. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Stacy R, Torrence WA, Mitchell CR. Perceptions of knowledge, beliefs, and barriers to colorectal cancer screening. J Cancer Educ. 2008;23(4):238–40. [DOI] [PubMed] [Google Scholar]
  • 5.Jimbo M, Rana GK, Hawley S, Holmes-Rovner M, Kelly-Blake K, Nease DE Jr, et al. What is lacking in current decision aids on cancer screening? CA Cancer J Clin. 2013;63(3):193–214. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Elwyn G, Scholl I, Tietbohl C, Mann M, Edwards AG, Clay C, et al. “Many miles to go…”: a systematic review of the implementation of patient decision support interventions into routine clinical practice. BMC Med Inform Decis Mak. 2013;13(2):S14. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Pieterse AH, Stiggelbout AM, Montori VM. Shared decision making and the importance of time. JAMA. 2019; [DOI] [PubMed] [Google Scholar]
  • 8.Brace C, Schmocker S, Huang H, Victor JC, McLeod RS, Kennedy ED. Physicians’ awareness and attitudes toward decision aids for patients with cancer. J Clin Oncol. 2010;28(13):2286–92. [DOI] [PubMed] [Google Scholar]
  • 9.Graham ID, Logan J, O’Connor A, Weeks KE, Aaron S, Cranney A, et al. A qualitative study of physicians’ perceptions of three decision aids. Patient Educ Couns. 2003;50(3):279–83. [DOI] [PubMed] [Google Scholar]
  • 10.Cornwall A, Jewkes R. What is participatory research? Soc Sci Med. 1995;41(12):1667–76. [DOI] [PubMed] [Google Scholar]
  • 11.Rycroft-Malone J, Burton CR, Bucknall T, Graham ID, Hutchinson AM, Stacey D. Collaboration and co-production of knowledge in healthcare: opportunities and challenges. Int J Health Policy Manag. 2016;5(4):221. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Golden-Biddle K, Reay T, Petz S, Witt C, Casebeer A, Pablo A, et al. Toward a communicative perspective of collaborating in research: the case of the researcher-decision-maker partnership. J Health Serv Res Policy. 2003;8(2_suppl):20–5. [DOI] [PubMed] [Google Scholar]
  • 13.Straus SE, Tetroe JM, Graham ID. Knowledge translation is the use of knowledge in health care decision making. J Clin Epidemiol. 2011;64(1):6–10. [DOI] [PubMed] [Google Scholar]
  • 14.Jagosh J, Macaulay AC, Pluye P, Salsberg J, Bush PL, Henderson J, et al. Uncovering the benefits of participatory research: implications of a realist review for health research and practice. Milbank Q. 2012;90(2):311–46. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Drahota AMY, Meza RD, Brikho B, Naaf M, Estabillo JA, Gomez ED, et al. Community-academic partnerships: A systematic review of the state of the literature and recommendations for future research. Milbank Q. 2016;94(1):163–214. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Gagliardi AR, Berta W, Kothari A, Boyko J, Urquhart R. Integrated knowledge translation (IKT) in health care: a scoping review. Implement Sci. 2015;11(1):38. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Marshall M, Pagel C, French C, Utley M, Allwood D, Fulop N, et al. Moving improvement research closer to practice: the Researcher-in-Residence model. BMJ Qual Saf. 2014;23(10):801–5. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Vindrola-Padros C, Eyre L, Baxter H, Cramer H, George B, Wye L, et al. Addressing the challenges of knowledge co-production in quality improvement: learning from the implementation of the researcher-in-residence model. BMJ Qual Saf. 2018;bmjqs-2017-007127. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Greenhalgh T, Wieringa S. Is it time to drop the ‘knowledge translation’metaphor? A critical literature review. J R Soc Med. 2011;104(12):501–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004;82(4):581–629. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Rowley E, Morriss R, Currie G, Schneider J. Research into practice: collaboration for leadership in applied health research and care (CLAHRC) for Nottinghamshire, Derbyshire, Lincolnshire (NDL). Implement Sci. 2012;7(1):40. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Graham ID, Kothari A, McCutcheon C. Moving knowledge into action for more effective practice, programmes and policy: protocol for a research programme on integrated knowledge translation. Implement Sci. 2018;13(1):22. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Hoekstra F, Ginis KAM, Allan V, Kothari A, Gainforth HL. Evaluating the impact of a network of research partnerships: a longitudinal multiple case study protocol. Health Res Policy Syst. 2018;16(1):107. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Gold M, Helms D, Guterman S. Identifying, monitoring, and assessing promising innovations: using evaluation to support rapid-cycle change, Issue Brief Commonw Fund. 2011;12(1):12. [PubMed] [Google Scholar]
  • 25.Ovretveit J, Hempel S,L Magnabosco J,S Mittman B,V Rubenstein L,A Ganz D. Guidance for research-practice partnerships (R-PPs) and collaborative research. J Health Organ Manag. 2014;28(1):115–26. [DOI] [PubMed] [Google Scholar]
  • 26.Ward M, De Brún A, Beirne D, Conway C, Cunningham U, English A, et al. Using co-design to develop a collective leadership intervention for healthcare teams to improve safety culture. Int J Environ Res Public Health. 2018;15(6):1182. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Schmittdiel JA, Grant RW. Crossing the research to quality chasm: a checklist for researchers and clinical leadership partners. Springer; 2018. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Lafata JE, Shin Y, Flocke SA, Hawley ST, Jones RM, Resnicow K, et al. Randomised trial to evaluate the effectiveness and impact of offering postvisit decision support and assistance in obtaining physician-recommended colorectal cancer screening: the e-assist: Colon Health study—a protocol study. BMJ Open. 2019;9(1):e023986. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.Alishahi AT, Fleming PJ, Shin Y, Resnicow K, Jones RM, Flocke SA, et al. Challenges and opportunities using online portals to recruit diverse patients to behavioral trials. J Am Med Inform Assoc JAMIA. 2019; [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Crowe S, Cresswell K, Robertson A, Huby G, Avery A, Sheikh A. The case study approach. BMC Med Res Methodol. 2011;11(1):100. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Stake RE. The art of case study research. Sage; 1995. [Google Scholar]
  • 32.King N, Template Analysis, in Qualitative Data Analysis in Organisational Research: A Practical Guide, Symon G and Cassel C, Editors. 1998, Sage: London. [Google Scholar]
  • 33.Denzin NK, Lincoln YS. The Sage handbook of qualitative research. Sage; 2011. [Google Scholar]
  • 34.Logic Models - Program Evaluation - CDC [Internet]. 2018. [cited 2019 Jun 5]. Available from: https://www.cdc.gov/eval/logicmodels/index.htm
  • 35.Budrionis A, Bellika JG. The learning healthcare system: where are we now? A systematic review. J Biomed Inform. 2016;64:87–92. [DOI] [PubMed] [Google Scholar]
  • 36.McGinnis JM, Aisner D, Olsen L. The learning healthcare system: workshop summary. National Academies Press; 2007. [PubMed] [Google Scholar]
  • 37.Weiner BJ. A theory of organizational readiness for change. Implement Sci [Internet]. 2009. [cited 2016 Oct 26];4:67. Available from: 10.1186/1748-5908-4-67 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38.Heaton J, Day J, Britten N. Inside the “black box” of a knowledge translation program in applied health research. Qual Health Res. 2015;25(11):1477–91. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 39.Heaton J, Day J, Britten N. Collaborative research and the co-production of knowledge for practice: an illustrative case study. Implement Sci. 2015;11(1):20. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40.Sibbald SL, Tetroe J, Graham ID. Research funder required research partnerships: a qualitative inquiry. Implement Sci. 2014;9(1):176. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 41.Institute for Healthcare Improvement: Science of Improvement: Forming the Team [Internet]. [cited 2019 Jun 17]. Available from: http://www.ihi.org:80/resources/Pages/Howtolmprove/ScienceoflmprovementFormingtheTeam.aspx
  • 42.Birken SA, Lee S-YD, Weiner BJ. Uncovering middle managers’ role in healthcare innovation implementation. Implement Sci. 2012;7(1):28. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 43.Carman KL, Dardess P, Maurer M, Sofaer S, Adams K, Bechtel C, et al. Patient and family engagement: a framework for understanding the elements and developing interventions and policies. Health Aff (Millwood). 2013;32(2):223–31. [DOI] [PubMed] [Google Scholar]
  • 44.The Value of Engagement | PCORI [Internet], [cited 2019 Jun 5]. Available from: https://www.pcori.org/about-us/our-programs/engagernent/public-and-patient-engagement/value-engagement
  • 45.Dyer KE, Shires DA, Flocke SA, Hawley ST, Jones RM, Resnicow K, et al. Patient-Reported Needs Following a Referral for Colorectal Cancer Screening. Am J Prev Med. 2019;56(2):271–80. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 46.Schuttenberg H, Guth H. Seeking our shared wisdom: a framework for understanding knowledge coproduction and coproductive capacities. Ecol Soc. 2015;20(1). [Google Scholar]
  • 47.Djenontin INS, Meadow AM. The art of co-production of knowledge in environmental sciences and management: lessons from international practice. Environ Manage. 2018;61(6):885–903. [DOI] [PubMed] [Google Scholar]
  • 48.Elston JL, Miller CA, Shires DA, Dyer K, Ratliff SM, Schreiber M. Patients’ adoption of and feature access within electronic patient portals. Am J Manag Care. 2018;24(11):e352–7. [PMC free article] [PubMed] [Google Scholar]
  • 49.Feldstein AC, Glasgow RE. A practical, robust implementation and sustainability model (PRISM) for integrating research findings into practice. Jt Comm J Qual Patient Saf. 2008;34(4):228–43. [DOI] [PubMed] [Google Scholar]
  • 50.Scott SD, Plotnikoff RC, Karunamuni N, Bize R, Rodgers W. Factors influencing the adoption of an innovation: An examination of the uptake of the Canadian Heart Health Kit (HHK). Implement Sci. 2008;3(1):41. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 51.Rogers EM. Lessons for guidelines from the diffusion of innovations. Jt Comm J Qual Patient Saf. 1995;21(7):324–8. [DOI] [PubMed] [Google Scholar]
  • 52.McLachlan S, Potts HW, Dube K, Buchanan D, Lean S, Gallagher T, et al. The Heimdall Framework for Supporting Characterisation of Learning Health Systems. J Innov Health Inf. 2018; [DOI] [PubMed] [Google Scholar]
  • 53.Nyström ME, Karltun J, Keller C, Gäre BA. Collaborative and partnership research for improvement of health and social services: researcher’s experiences from 20 projects. Health Res Policy Syst. 2018;16(1):46. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 54.Montana J Co-production in action: perceiving power in the organisational dimensions of a global biodiversity expert process. Sustain Sci. 2019;1–11. [Google Scholar]
  • 55.Miller CA, Wyborn C. Co-production in global sustainability: histories and theories. Environ Sci Policy. 2018; [Google Scholar]
  • 56.Greer LL, van Kleef GA. Equality versus differentiation: The effects of power dispersion on group interaction. J Appl Psychol. 2010;95(6):1032. [DOI] [PubMed] [Google Scholar]
  • 57.Greer LL, Van Bunderen L, Yu S. The dysfunctions of power in teams: A review and emergent conflict perspective. Res Organ Behav. 2017;37:103–24. [Google Scholar]
  • 58.Tabriz AA, Birken SA, Shea CM, Fried BJ, Viccellio P. What is full capacity protocol, and how is it implemented successfully? Implement Sci. 2019;14(1):73. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 59.Caldwell DF, Chatman J, O’Reilly III CA, Ormiston M, Lapiz M. Implementing strategic change in a health care system: The importance of leadership and change readiness. Health Care Manage Rev. 2008;33(2):124–33. [DOI] [PubMed] [Google Scholar]
  • 60.O’Reilly CA, Caldwell DF, Chatman JA, Lapiz M, Self W. How leadership matters: The effects of leaders’ alignment on strategy implementation. Leadersh Q. 2010;21(1):104–13. [Google Scholar]
  • 61.Baker GR, Denis J-L. Medical leadership in health care systems: from professional authority to organizational leadership. Public Money Manag. 2011;31(5):355–62. [Google Scholar]

RESOURCES