Team Science Approach to Developing Consensus on Research Good Practices for Practice‐Based Research Networks: A Case Study

Abstract

Using peer learning strategies, seven experienced PBRNs working in collaborative teams articulated procedures for PBRN Research Good Practices (PRGPs). The PRGPs is a PBRN‐specific resource to facilitate PBRN management and staff training, to promote adherence to study protocols, and to increase validity and generalizability of study findings. This paper describes the team science processes which culminated in the PRGPs. Skilled facilitators used team science strategies and methods from the Technology of Participation (ToP®), and the Consensus Workshop Method to support teams to codify diverse research expertise in practice‐based research. The participatory nature of “sense‐making” moved through identifiable stages. Lessons learned include (1) team input into the scope of the final outcome proved vital to project relevance; (2) PBRNs with diverse domains of research expertise contributed broad knowledge on each topic; and (3) ToP® structured facilitation techniques were critical for establishing trust and clarifying the “sense‐making” process.

Introduction

Practice‐based research networks (PBRNs) are networks of primary care, community‐based practices devoted to asking and answering questions relevant to the patients who attend these practices. Since the early 1970s, PBRNs have been instrumental to the translation of clinical knowledge, medical discoveries, and healthcare innovations into primary care practice.1, 2 One of the hallmarks of PBRN research is the multidisciplinary nature of research conducted in geographically dispersed locations to address complex clinical practice challenges. Team‐based research is critically important to addressing complex health problems3 and is necessary if primary care PBRNs are to accelerate adoption, implementation, and sustainability of clinical practice improvement.

Team research leadership and engagement strategies maximize, “interdependence, joint ownership, and collective responsibility between and among scientists”.4 Therefore, the goal of this project was to apply team science strategies to engage the directors, managers and staff of nationally recognized PBRNs to translate practical research knowledge and experience into a compendium for community and practice‐based research networks. Team science strategies for this project used conceptual and methodological strategies to initiate the work and finalize the product. The processes used in this study contribute to a better understanding of how teams exchange expert knowledge, bring diverse viewpoints to consensus, and engage external stakeholders in dissemination. This paper describes five developmental phases that produced the Practice‐based Research Good Practices (PRGPs) and refined leadership skills in team‐based communication. The PRGP document, a compendium of recommended strategies developed by experienced primary care researchers that covers four domains (Building PBRN Infrastrucure, Study Development and Implementation, Data Management, and Dissemination Policies) is described elsewhere5 and is also available at: http://www.napcrg.org/PBRNResearchGoodPractice.

Preproject Background

This project built upon a preliminary work to establish a foundation for improving PBRN research processes.6, 7, 8, 9 An earlier preliminary study surveyed 75 PBRN directors/staff to identify and rank key research practices essential for sound PBRN research practice. Using a factor‐analytic approach, 31 research best practices loaded on five domains (PBRN management, study supervision, data management, PBRN policies, and ethical considerations) and a self‐assessment checklist was developed for PBRNs to assess existing research procedures. The current study sought to address the gap in knowledge regarding PBRN‐specific research practices and implemented a highly collaborative, integrated approach engaging multiple disciplines and clinician researchers from PBRNs with unique histories and areas of expertise. Study participants had no previous history of previous direct research collaboration. The purpose of this case study is to demonstrate qualitative approaches that were effective in building a “team science” collaboration, and to describe strategies essential to the success of the project (flexible adaptation to team goals; facilitated communication) to stimulate knowledge sharing and shared agreement on what constitutes “research good practices.”

Project Approach

Practice‐based research networks

Practice Based Research Networks (PBRNs) with an established track record of leadership in research and network engagement were recruited for this project. The seven PBRNs that agreed to participate were (1) the Duke Primary Care Research Consortium (PCRC);10, 11, 12 (2) the Iowa Research Network (IRENE);13, 14, 15 (3) the Metropolitan Detroit Practice‐based Research Network (MetroNet);16, 17, 18 (4) the Oklahoma Physicians Resource/Research Network (OKPRN);19, 20, 21 (5) the Oregon Rural Practice‐based Research Network (ORPRN);22, 23, 24 (6) Research Involving Outpatient Settings Network (RIOS Net);25, 26, 27 and (7) Wisconsin Research and Education Network (WREN).28, 29, 30 Project leadership and administration was provided by the MetroNet director and research faculty.

The initial plan was to produce a subset of standard operating research procedures (SOPs) using the Model for Improvement31 and Plan‐Do‐Study‐Act (PDSA)32 cycles to test and refine research recommendations. However, early on the teams concluded that the planned testing cycles were logistically difficult and incompletely captured the complexity of PBRN research practices. The teams agreed that a greater value for PBRNs would be a comprehensive document of recommended research practices specific to the special conditions of the decentralized community‐based clinical practice setting. Experts envisioned a sustainable, living document that captured excellence in current research practice and anticipated ongoing updates as the field evolved. The Model for Improvement and PDSA were thus omitted as a process for the endeavor.

Team Composition

The six PBRNs were divided into two teams involving a network director and one or two research personnel from each PBRN. Teams were grouped according to director interests and network priorities identified on a baseline assessment of current research practices. Each PBRN used the Research Best Practices Checklist to review their current research procedures and identify areas for improvement.6, 7, 8, 9 Interest in improving data management was a top priority for all six networks. Teams agreed to elaborate different aspects of the data management section.

Team Interaction

Over the 3 years of the funding period, teams interacted in a series of five biannual in‐person team meetings, supplemented with monthly team webinars and independent writing. Depending on the subject matter and interests, team members worked independently, with their teams, and with the combined team group to propose, discuss, and iteratively elicit input from their PBRN staff members regarding current research procedures. Discussion was guided by two experienced group facilitators from the University of Calgary (PS and MO) who modeled, reinforced, and mentored team members as group facilitation leaders. Team composition remained stable until the final year when working groups restructured based on content expertise and interest. Attendance and participation at these five in‐person meetings was stable over the 3 years, suggesting the commitment of the participants to the project.

Group Facilitation

In order to build trust, we employed group facilitation strategies through the entire project. The Technology of Participation (ToP®) was the group facilitation tool used in this program and ultimately team members were trained in its application. ToP® consists of methods that enable groups to (a) engage in thoughtful and productive conversations; (b) develop common ground for working together; and (c) build effective short‐ and long‐range plans.33 The two primary strategies used in ToP® are the (1) Focused Conversation Method34 and the (2) Consensus Workshop Method.33 The Focused Conversation Method component enables a conversation to flow from surface‐level facts to more in‐depth personal beliefs about a topic using a series of questions organized into a typology to direct groups towards making a decision,33, 34, 35, 36 where the following process is completed: Objective (directly observable facts); Reflective (emotions and feelings associated with the facts); Interpretive (what is significant or meaningful); and Decisional (decisions or actions to be taken).

The Consensus Workshop Method helps a group form a working consensus as it discovers the common ground needed to move forward. 33 This method was used in the five in‐person meetings held during the first three project years. Using a Socratic method that seeks multiple agreed‐upon answers to questions and allows groups to generate and consider a number of possible answers, the method facilitates groups to achieve agreement on several points.36 The five‐step process is: (1) context setting (setting the stage); (2) brainstorming (objective, generates new ideas); (3) grouping (reflective, forming new relationships); (4) naming (interpretive, discerning the consensus); and (5) reflecting (decisional, confirming the resolve). The facilitators used this method to help the groups achieve consensus on the general outline of good research practices and strategies for individual research practices. The professional facilitators attended each of the in‐person meetings and used these team science strategies to meet the meeting objectives.

Analysis of Project Process

The teams’ commitment expanded into a fourth year, and a retrospective, qualitative analysis of the project's process records was conducted. Process records are detailed in Table 1. Documentary evidence were assembled and organized sequentially. Team dynamics were captured in meeting minutes, observer field notes, and postmeeting debriefings, as an inductive process structured the content of effective research practice (“new knowledge”). Teams debated research practices judged to be foundational (essential) as opposed to specialized (optional). The organization of the final document continuously evolved based on the structure of the “new knowledge” and changing expectations of end‐users. Sentinel transitions in design and content of the final compendium were used to mark phase changes. Interpretive analyses were reviewed and validated by project members. These qualitative approaches supported group consensus and maintaining participant commitment.

Table 1.

Qualitative data document sources used for process evaluation (collected 2011–2014)

Source document types	N
1. Research practice self‐assessments (4 cycles, 6 PBRNs)	24
2. Team meeting: observer field notes & debriefing summaries (onsite; web‐based meetings)	130
3. Team meetings: facilitator process guides (onsite meetings)	4
4. Team meetings: participant evaluations (onsite)	8
5. Evaluation team meetings: agendas, minutes, e‐discussions	25
6. Workgroup meetings: SOP development (agendas, minutes, e‐discussion notes)	16
7. Leadership team meetings: agendas, minutes	>100
8. Annual reports to funder	4
9. Peer review comments: preconference workshops, national presentation; P30‐Wiki editors: emails, document reviews	>100
10. PBRN research procedures document: iterative drafts over 4 years	Ongoing; continuous

Development of “New Knowledge”

Five stages were identified in the development of the PRGPs (see Table 2). In Stage 1, a baseline understanding of the research histories and current priorities and interests of each PBRN network was used to compose teams with diverse histories but shared needs and interests. All PBRNs gave highest priority to developing procedures for specific Data Management practices. Teams selected one research practice in the Data Management domain as a priority, either “The PBRN uses established protocols to promote data accuracy” or “The PBRN documents all steps in the data management process.” This stage set the initial target for team activities.

Table 2.

Project stage by primary strategy used to develop research good practices

	Stage 1 (Year 1)	Stage 2 (Early Year 2)	Stage 3 (Late Year 2)	Stage 4 (Year 3)	Stage 5 (Years 4 and 5)
Team activities (process evidence)	Establish baseline research procedures PBRNs Assess existing SOPs for 31 best practices (BPs) Identify network priorities Compose teams	Specify and test a procedure for a segment of one BP. List, cluster, name & define key components Consensus on typical workflow Metrics	Develop procedures for multiple BPs Discover and reconcile procedural gaps, overlaps, and redundancy Categorize and quantify BPs Reduce to 12 major BPs	Move to a more comprehensive document of research “procedures” Add new chapter: infrastructure Develop examples, templates, and resources	Confirm practical relevance of document Expand content on: dissemination practices; research procedures
Team discussion focus (narrative evidence)	“How do we currently assure quality research practice in our network?” Define current research practices	“What steps make this happen?” Inductive use of workflow diagrams to break practice into discrete parts Build the “ideal” practice	“We notice that research practices are interconnected. Reorganization is needed.” “We feel more comfortable with the inductive process.” Pattern recognition Efficiency accelerates	“There are gaps and redundancies when we compare this to our actual practice.” Move away from “BP‐centric” procedures Realign “ideal” with actual network management practices	“Are these guidelines useful and accessible to the PBRN community?” • PBRN community reviews and validates final document

In Stage 2, groups initiated early collaboration activities led by the group facilitators who guided teams in a series of cognitive exercises to specify the content and processes for the assigned best practice. Content was organized and defined, and examples developed for each element to support the shared understanding across the team. PBRN members differentiated elements considered as “required (core)” from those viewed as “optional (as needed)” depending upon study circumstances, network resources and study design. PBRN members worked independently and presented to the team at monthly webinar/conference calls. In this stage, the work focused on exchanging PBRN‐specific practices, gaining understanding about similarities and differences between networks, and inductively constructing an “ideal” research procedure. Trust developed slowly in this stage as network directors discussed areas of research strength as well as challenges. Trust evolved as practical experience was shared, for example the exchange of internal tools, procedures and templates. Team operational rules evolved in this phase. The administrative team clarified communication rules to support team effectiveness, for example, email reminders, team calendars, travel planning, webinar resources.

In Stage 3, teams initiated a reconsideration of the original goal which shifted from developing and testing discrete segments of a single Data Management practice to a focus on an entire domain, for example, all the best practices associated with effective Data Management. Teams began to notice patterns and noted instances when procedures in separate practice areas were related and interdependent. Reorganization of the document structure was necessary. Some research practices were realigned as subareas of larger research practices. Teams reported feeling more comfortable with the consensus process (inductive approach) and elected to accelerate the project timeline to address multiple research practices within an entire practice domain. Teams moved away from the initial idea to “develop and objectively test” individual procedures. The procedures for the Data Management section developed independently by each team were combined and reorganized to eliminate redundancies and to anticipate the research needs of PBRNs at all developmental stages, now an organizing theme. Enthusiasm for the project was great, and teams elected to commence the development of two additional domains, Study Supervision and PBRN Publication Policies while finishing work on Data Management assignments.

In Stage 4, a second “goal‐shift” occurred when teams decided that a comprehensive documentation of research procedures was the desired outcome. Team consensus discussions continued a conceptual shift from “best practice‐centric” to a “good practices” orientation which flowed from team discussions of research practice “excellence” under diverse PBRN conditions and recognition that empirical support was not sufficient to support the use of a “best practice” terminology. As teams debated and discussed, project leadership listened and translated conceptual shifts into the organizational structure of the emerging document. For example, Research Ethics, originally a stand‐alone topic, was moved into Study Supervision. The PBRN Policies section was clarified to address publications and presentations. Onsite Study Supervision was differentiated from (offsite) Study Management and retitled based on locus of study leadership and clinical oversight. A new domain, PBRN Infrastructure was championed by one team in order to address the essential human and material resources necessary to enable rigorous PBRN research practices and an important discussion about leadership accountability for staff development ensued. At this stage, references to SOPs were dropped and replaced with a generic term, “research procedures.”

In the final Stage 5, a comprehensive draft was composed, teams merged and reorganized into partnerships based on practical research expertise for the purpose of polishing the final section drafts. The document structure and content was also subject to multiple cycles of internal and external peer review by the wider PBRN community. Strong guidance was received from researchers affiliated with PBRNs at all developmental stages as the draft was disseminated and discussed at interactive sessions at professional meetings.37, 38, 39, 40 Suggestions were incorporated into significant revisions to each chapter. For example, after the North American Primary Care Research Group (NAPCRG) PBRN 2013 workshop,41 the “Publication and Presentation Policies” chapter was renamed and significantly expanded to better address the critical obligation to disseminate PBRN research to a broad clinical and community audience.

The penultimate PRGP document draft was subject to external review by members of the PBRN workgroup of the NAPCRG Committee on the Advancement of the Science of Family Medicine (CAFSM), along with volunteers affiliated with three of the AHRQ‐funded PBRN Centers of Practice‐based Research and Learning.41

After the final round of peer‐review, the completed PRGP document was introduced to a national audience during AHRQ‐sponsored webinar.42

Discussion

In this project a “knowledge sharing” model developed by Snowden43 is useful for examining how the practical knowledge of PBRN research experts was a surfaced, specified, and ordered into the PBRN Research Good Practices document. This model describes team growth process and may be of use to other nascent collaboratives of interdisciplinary scientists.

Practice‐based research knowledge, given its location in dynamic primary care contexts with diverse populations, has large domains that are not entirely “known” (predictable; well‐defined; causal). Thus, the effort to compose an initial set of PBRN research procedures, or to “make sense” of what is well known to PBRN experts, must by necessity employ a decision‐making approach that is reflects reliability of available evidence, divergent perspectives and alternative sources of “new knowledge.”

Snowden's “Cynefin Framework”43, 44, 45 specifies domains of “collective sense‐making” based on the degree of uncertainty, order and knowability of available information. The domain where certainty is maximal, the “Known” domain, occurs when knowledge is well supported empirically with a sufficient evidence base to justify “best practices.” While reference to Standard Operating Procedures and Best Practices was familiar language and facilitated discussion, teams soon acknowledged the contextual nature of PBRN research knowledge and the lack of “hard science” on research practice in the pragmatic setting of decentralized ambulatory practices. More often teams were working in knowledge areas that lacked empirical support, but where they brought considerable experiential wisdom established through years of professional practice. Snowden44 terms this the “Knowable domain.” Empirical support could be generated, but is not yet available and decisions are based primarily on experiential knowledge. This domain is evidenced in the semantic changes as the project moved away from terminology such as “standard operating procedures” and “best practices” in recognition that “guidelines” and “good practices” were more accurate. This was the dominant level of knowledge informing the PRGP document.

Alternatively, knowledge based less on practical experience and more on abstract ideas, emerged in collegial brainstorming or discussion of hypothetical conditions. Snowden44 terms the “Complex domain” in which uncertainty increases but potential answers can be generated through collegial exchange. It is in this domain where complexity science is located. In this project, teams strayed into this domain when faced with the interconnections and contingencies of PBRN research surfaced in Stage 3 and teams searched for a new structure and organization for the PRGP document.

At intervals, sense‐making did descend into disorder without clear consensus or shared experience (Chaos domain) but these intervals were brief and became less disturbing as the teams developed confidence in the process, and trust among their new colleagues. There was a recognition that an inductive process of open thinking, while at times chaotic, could ultimately move the teams to shared recognition of what good research practice could “look like” (Complex domain), and ultimately community consensus on the content and structure of this initial version of the “PBRN research good practices” document (Knowable domain). The work was not linear or always comfortable, particularly when order collapsed or practical knowledge was questioned. PBRN research knowledge is not unlike medical knowledge,46 in that it is inherently uncertain and dependent on context and experience, for example in anticipating the changing conditions of working in geographically dispersed, primary care practices.

Team Science in the Production of the PRGP Document

Participatory goal setting (shared aspirations; articulating the desired goal)

A key lesson learned in this project was that end‐users involved in a study relevant to their needs and interests will engage and persist despite multiple barriers. As the teams clarified and transformed the scope and content of the document, their commitment and persistence carried them through and beyond the original 3 years of the project. In fact, teams voluntarily expanded the scope of the project into a fourth year because it addressed an important gap in PBRN research practice, and they believed that they were working for the common good, to improve primary care research practice. The PRGP document's organizational frame and content (which included examples and resources) moved consistently toward the practical needs of PBRNs.

Getting through the rocky periods (trust; communication; handling conflict)

This project affirmed that team effectiveness and ultimately the quality of the final PRGP document, was contingent on establishing solid interpersonal relationships and open communication and accommodation of diverse perspectives. Over time the team's comfort working in uncharted territory increased. The structured ToP®33, 34 group facilitation techniques were essential for building trust and clarifying the “sense‐making” process for team members. It was this active, guided process that “bent the arc” of this project; that broadened and codified a product based on the team's professional expertise and commitment to developing practice‐based research procedures suited to the realistic needs of PBRN researchers. The teams cycled between inductive work and deductive work as they noticed cross‐linkages, gaps, and redundancies across research practices. Informal knowledge based on professional expertise of “what works” anchored months of discussion among PBRNs with different research portfolios and organizational structures. Critical to this work were early and ongoing team‐building efforts that solidified professional relationships and collaborations that continue today. The process ultimately redefined the final outcome as teams moved beyond the original project goals toward a comprehensive, practical compendium of recommendations for PBRNs at all stages of development and in diverse settings and configurations.

“Landing the fish” (appreciative inquiry; capacity building; sustaining/building forward; dissemination; ongoing partnerships)

As the end of the funded project period approached, teams sensed that time was running out. Consultants quickly responded by convening a “Strategic Planning Workshop” during the final face‐to‐face meeting. This empowered teams to define the terms of an acceptable closeout and the steps to ultimately “landing the fish.” Appreciative inquiry was another technique to support an accounting of what had been accomplished and the successful development of collaborative leadership skills. Finally, respect for the fragility and contested nature of the PBRN research supported the view that the PBRN Research Good Practices is a “living document” that will need ongoing revision as research practice evolves.

Limitations

The PRGPs are limited in scope. Areas purposively not addressed were research study design, writing proposals and responsible conduct of research, as these areas seem well supported by ample resources. In addition, determining best practices for a library of novel research methods would require substantial time and resources which were not possible with the available resources. Instead, the group refocused its approach to a more practical agenda: to delineate research practices that have a proven track record in advancing PBRN research and producing good outcomes.

Conclusion for Team Science Approaches

The purpose of this project was to detail recommended research practices specific to the context of practice‐based research networks by synthesizing collective knowledge of respected network directors and research staff. The team‐based process experienced in this project tracks closely with current developments in team science.47, 48, 49 A framework of “sense‐making”45 best captured the dynamic ebb and flow of certainty and uncertainty as consensus slowly developed around a new set of research practices based on practical wisdom of experts in the field. Certainly not a linear process, new knowledge increased in rigor and formality in a series of iterative cycles that spiraled from inside the project to the wider PBRN community and back again as clarity increased. Diverse practical perspectives about “what works” in the PBRN research environment were ordered and reorganized and vetted by the wider PBRN community. Arriving at a compendium of good practices for practice‐based research itself needed a practical, practice‐oriented, and a more “end‐user driven” participatory approach. This process produced a document that had a high added value and improved acceptability by the intended audience. Even after the funding period ended for this demonstration project, the teams persisted in this work, and in the fifth year are now developing a new set of recommended strategies related to stakeholder engagement.50, 51

Conflicting and Competing Interests

The authors have no conflicting or competing interests to declare.