Uncovering the wisdom hidden between the lines: the Collaborative Reflexive Deliberative Approach

Abstract

Background

Meta-analysis and meta-synthesis have been developed to synthesize results across published studies; however, they are still largely grounded in what is already published, missing the tacit ‘between the lines’ knowledge generated during many research projects that are not intrinsic to the main objectives of studies.

Objective

To develop a novel approach to expand and deepen meta-syntheses using researchers’ experience, tacit knowledge and relevant unpublished materials.

Methods

We established new collaborations among primary health care researchers from different contexts based on common interests in reforming primary care service delivery and a diversity of perspectives. Over 2 years, the team met face-to-face and via tele- and video-conferences to employ the Collaborative Reflexive Deliberative Approach (CRDA) to discuss and reflect on published and unpublished results from participants’ studies to identify new patterns and insights.

Results

CRDA focuses on uncovering critical insights, interpretations hidden within multiple research contexts. For the process to work, careful attention must be paid to ensure sufficient diversity among participants while also having people who are able to collaborate effectively. Ensuring there are enough studies for contextual variation also matters. It is necessary to balance rigorous facilitation techniques with the creation of safe space for diverse contributions.

Conclusions

The CRDA requires large commitments of investigator time, the expense of convening facilitated retreats, considerable coordination, and strong leadership. The process creates an environment where interactions among diverse participants can illuminate hidden information within the contexts of studies, effectively enhancing theory development and generating new research questions and strategies.

Background

Science advances carefully in small increments with rare major breakthroughs punctuating an otherwise methodical research and knowledge trajectory. Breakthroughs often follow a period where a field of study appears stuck. Is there a way to accelerate the process of getting unstuck? Investigators working on similar content within different contexts traditionally learn from each other through reading published manuscripts, talking at conferences, and participating together on committees and in groups. However, the purpose of these activities is to discover and discuss key findings and not, in any systematic way, understand the experiential and deeper tacit knowledge of processes and contextual factors. Thus, there is a tendency to ‘recreate the wheel’, in part because investigators are unable to meaningfully share and sense-make across their respective contexts.

In recent years, systematic approaches and techniques have been developed to quantitatively synthesize results across published studies (i.e. meta-analysis). Similar approaches (meta-synthesis and meta-ethnography) for qualitative research have also emerged (1,2). Narrative synthesis, an extension of meta-synthesis, seeks to identify a coherent story instead of themes (3). Recently, two related approaches have been developed for reviewing and synthesizing mixed-methods evidence. Meta-narrative synthesis explores underlying assumptions and paradigms within study contexts (4), while realist synthesis is a more generative approach than the content analysis-like strategy of meta-synthesis (5). Though realist perspective does not start as much with a priori themes (6); it is still largely grounded in what is already published, missing the tacit ‘between the lines’ knowledge generated during many research projects and not intrinsic to the main objectives of studies.

While the science of research synthesis has grown, researchers face challenges when trying to pool and compare studies derived from differing methods, paradigms, and contexts. Often the research questions used in studies were not aligned with the questions of a comparative inquiry in mind. Additionally, none of the existing strategies deliberately engage the original researcher or ‘knowledge generator’ in the meta-process. Thus, experienced researchers might benefit from a methodology using an interactive process of systematic reflection and synthesis that uncovers details and potential wisdom from their own work as well as other investigators’ research in their area.

The purpose of this manuscript is to describe a methodology for generating new information (capturing ‘between the lines’ knowledge) by catalysing interactions among researchers to generate additional knowledge across boundaries in a field of study. The example, here, is the delivery of primary health care services using teams, but the method may be used for any focus. We developed this methodology, which we term the Collaborative Reflexive Deliberative Approach (CRDA), to uncover and share tacit knowledge, expand and deepen understanding of context, reveal and challenge hidden assumptions and make better sense of variations from a collection of similar quantitative, qualitative and mixed-methods studies. Furthermore, we sought to develop a methodology that pools streams of research within a common area to address underlying problems related to our research questions and approaches by sharing published and unpublished literature and the lived experiences of investigators by explicitly not keeping original study authors at a distance.

CRDA draws upon the research synthesis approaches mentioned above combined with some principles of participatory action research (7). Participatory action research integrates participatory interaction and the lived experience of action into the research enterprise. In our approach, established investigators were brought together as active observers and participants in a deliberative iterative process of sharing, reflection and synthesis. Deliberative process allows a group of participants to receive and exchange information, critically examine an issue, and come to consensus agreement or clear understanding of disagreements. Specifically, an analytic-deliberative approach was used which combines technical and knowledge expertise with stakeholder (i.e. investigators) values and experiences. It includes creating a safe space, engaging everyone, and using models of democratic governance and consensus-decision making (8). CRDA distinctively enables investigators from different geographic and policy contexts to come together as active participants exploring and challenging each other’s published findings, underlying assumptions and experiential knowledge, while also reflecting together on insights, experiences, and data that go beyond what has been published.

Method

Getting started

The ideas behind CRDA began at a conference with three experienced researchers discussing the possibility of combining data from multiple studies they had led. They were very familiar with each other’s work and recognized that they needed to design a methodology of getting beyond what had been published to discover what was hidden ‘between the lines’. Because one was from Australia, one from Canada and another from the USA, they struggled to come up with a strategy for bridging distinct contexts while taking advantage of the commonalities in their research. A Canadian Institute of Health Research (CIHR) Catalyst Grant provided the impetus for these investigators to invite additional colleagues with common research interests to form a new research collaboration in primary and community-based health care focusing on primary care and health system reform. The collaboration added eight established investigators with relevant reputable track records in publications and representing three provinces in Canada, Australia and the USA. Although two members of this expanded team worked in policy, a single research savvy decision-maker was recruited and participated in all activities to continually remind the group of key policy imperatives. The resulting team was knowledgeable about the studies they had conducted, experienced in a particular content area, and had ownership of the original data (Table 1). The goal was to have sufficient diversity among studies and collaborators, while still being manageable.

Table 1.

Overview of 12 primary care reform studies

Study title	Context	Study description
Re-order—Re-organizing the care of depression and related disorders in a primary care setting	Victoria and Tasmania, Australia	The re-order project was a longitudinal observational and participatory action research project set in six practices in Victoria and Tasmania from 2005 to 2008. The aim was to gather information to assist in the design of a new model for thinking about, and improving, primary care depression diagnosis and management. Phase 1 and 2 gathered the views of stakeholders about the key elements of exemplary depression care—576 patients and 300 stakeholders from clinical, academic, public and policy settings. The third phase involved working with general practices to document depression care in the Australian setting to identify areas for improvement, test out interventions for improving and develop principles for an exemplary model of depression care for Australia. (9,10)
Teamwork	Three states—Australia	The Teamwork study was a large cluster randomised controlled trial involving 60 practices in three Australian states. The intervention involved facilitation of teamwork in chronic disease management involving staff collocated within existing practices. This was relatively effective in developing collaborative activities especially care planning and shared information systems, and some improvements in practice routines. These were more effective in small practices. There was improved trust but the roles of nurses were still underdeveloped. (11,12)
Teamlink	New South Wales, Australia	The Teamlink study was a quasi-experimental study in 34 practices in one Australian state. The intervention aimed to increase teamwork between general practice and allied health providers located outside the practice. The structural links were provided by the requirement that referral to allied health required a GP care plan to specify which providers were involved in the ‘team care arrangement’. In response to facilitation, there was evidence of improved referrals but there was little progress in developing trust, effective direct communication and power sharing. (13,14)
Prac-Cap	Australian states and one territory	The Practice Capacity Study was a cross-sectional mixed methods study of the capacity of Australian general practice to provide multi-disciplinary planned care for patients with chronic conditions. It was conducted in New South Wales and South Australia across six primary care organizations in the context of the introduction of support for care planning and multidisciplinary care through Australia’s national health insurance scheme. Methods included surveys and qualitative interviews with medical and non-medical staff in general practice. The study found consistent themes including the tension between patient and practitioner’s goals for care, challenges with time management, the need for better systems to support team based care and care planning. (15,16)
CoMPaIR—Strengthening primary health care services through innovative practice networks	Alberta, Canada	CoMPaIR was a longitudinal, participatory, deliberative program of research using a cross-case comparative design to develop in-depth understanding of the interrelationship between context and models of primary care and their impact on interprofessional relationships. One specific intent was to support capacity development for sharing and using evidence among study participants. The program was implemented in two phases—local and provincial. The research team worked with local leaders to identify a particular program or project on which to focus. Three Primary Care Networks (PCNs) located within the former Calgary Health Region participated in phase 1; two additional PCNs participated in Phase 2. All five participating PCNs were mandated to achieve five common objectives. Despite this provincial commonality, local context had a marked influence on the models that were adopted and the ways in which teams functioned. A final component of the study involved comparison of the results from phases 1 and 2 with similar studies in other provincial contexts. (17)
COMP-PC—Comparison of models of primary health care in Ontario	Ontario, Canada	The comparison of models was a cross-sectional observational study of four family practice models in Ontario during a transformative change period. The study found that no one model that was superior in all aspects of quality. There were large variations in the quality of care between practices of the same model, and several factors were found to be more strongly associated with the quality of care delivered than the model itself. These factors included practice organization and team structure. (18–20)
Behind the Closed Door—using ethnography to understand family health teams	Ontario, Canada	This study, set in Ontario investigated the effect of the implementation of an advanced primary health care delivery model, the Family Health Team (FHT), on organizational and clinical routines, particularly those relating to the care of persons living with chronic illness. The study found wide variability in the implementation of chronic disease management. Several of the FHTs were grounded in traditional routines, making little use of new approaches to care delivery. In those FHTs where these routine changes took hold, a significant change was triggered in the physicians’ routines, facilitated by collaborative leadership and a history of reform within the practice. Existing physician oriented incentive structures provided subtle barriers to inter-professional care. (21–23)
Accessibility and Continuity of Care: A Study of PHC in Québec	Quebec, Canada	This study looked at various organizational models of primary care and their influence on accessibility and experience of care users. The various models related to differential level of teamwork being promoted by the primary care reform efforts. The models implemented involved mostly teams of doctors and nurses working together, linked by a formal contractual agreement within the practice and with local health authorities, and supported by governmental grants to fund administrative and rostering tasks. (24,25)
MaChro-1—primary health care models for patients with chronic disease	Quebec, Canada	This study looked at various organizational models of primary care and their influence on health, utilization and self-care for a cohort of chronically ill patients. The various models related to different levels of teamwork as part of the primary care reform. (26,27)
Prevention and Competing Demands in primary care	Nebraska, USA	Ethnographic descriptive study of 18 practices to understand variation in quality of care. The Prevention and Competing Demands was a descriptive study using in-depth case studies of family medicine practices and discovered little evidence of teamwork in the delivery of preventive services. This led to the design of the Using Learning Teams for Reflective Adaptation or ULTRA intervention study. (28–30)
ULTRA—Using Learning Teams for Reflective Adaptation	New Jersey and Pennsylvania, USA	Practice intervention in 56 primary care practices using facilitated team-building and reflection to enhance quality of care. The ULTRA intervention study which specifically targeted the development of communication and teams using a reflective adaptive process or RAP to enhance quality of care. Despite not having regular practice meetings at baseline, 18 of 25 practices successfully convened improvement teams. There was evidence of improved practice-wide communication in 12 of these practices if strong leaders were involved. Eight practices continued RAP meetings for 2 years and found the process valuable in problem solving and decision-making. (31–34)
NDP—National Demonstration Project (NDP)	USA	Multi-method evaluation of the first major implementation of the Patient-centred Medical Home in the USA. The NDP was launched in June 2006 as the first national test in the United States of a model of a particular PCMH model in a diverse sample of 36 family practices. NDP practices made substantial progress toward implementing the technical components; however, there was little evidence that practices actually changed their work relationships. It was apparent that for most practices the process will take a high degree of motivation, communication and leadership; considerable time and resources; and probably some outside facilitation. (35–40)

The CIHR grant provided modest funding for two face-to-face meetings, research coordination and teleconferences.

Teleconferences

The first several months of our CRDA were spent using conference calls to forge links among team members. These initial conference calls were used to decide on and discuss the studies to be included and to finalize the agenda and logistics for the first face-to-face meeting.

Face-to-face meetings

The first face-to-face meeting for CRDA needed to set the initial conditions for future meetings and expectations for future conversations. An experienced facilitator was hired to facilitate and to help create an environment for the collaborators to get better oriented to each other’s work and perspectives. The facilitator gained agreement on ground rules for interaction and led the collaboration towards developing focus for the work. Subsequent conference calls and follow-up face-to-face workshops led to the development of cross-project insights and conceptual models.

Collaborative reflexive deliberative approach

CRDA involves iterating through four fairly distinct yet inter-dependent phases (A, B, C and D in Figure 1), with new information added and processed with each phase (Table 2). Figure 1 diagrams the overall CRDA methodology, with sources of data and types of knowledge in the circles. Key tasks that iteratively take place with each type of data are presented as curly brackets. The following describes these four phases and the information, knowledge used.

Figure 1.

Diagram of the collaborative reflexive-deliberative approach

Table 2.

Detailed outline of the Collaborative Reflexive Deliberative Approach method

A. Refining peer publications
Occurs locally and via teleconferences and e-mail	1. Select
	i. Select diverse membership of collaborative senior investigators
	ii. Original published peer reviewed research relevant to broad aim
	iii. Fill in overview templates for each publication and distribute to all
Occurs at face-to-face meeting	2. Focus
	i. Discuss overviews
	ii. Brainstorm, pile sort, concept map to focus and refine research question
	3. Select—review and select publications for re-focused synthesis goal.
	4. Extract
	i. Create matrix
Occurs locally and via teleconferences and e-mail	ii. Fill in matrix from publication
B. Organizing and adding broader study materials
	1. Classify—determine levels of evidence
	2. Re-Extract—add broader materials (unpublished data, unanalysed data, interviews with others on the original study teams) into the matrices
	3. Analyse matrix data with frequent adjustments
At least one face-to-face meeting towards end of iterations	4. Iterate above multiple times till ‘feel stuck’—clarify terms and definitions
C. Interpreting and adding external collaborative knowledge
Occurs locally and via teleconferences and e-mail	1. Analyse—break into small groups for in-depth analysis
	2. Absorb—enter knowledge from participation in other potentially related collaborative projects into small group work
	3. Re-Interpret
	i. On-going interpretation till all pertinent data entered into refined matrices
	ii. Each person reviews all the matrices and creates summary of findings
D. Integrating and adding experiential reflections
Occurs at face-to-face meeting	1. Re-Interpret—individual summaries shared with discussion
	2. Reflect—adding experiential knowledge within group process
	3. Integrate
	i. Incorporating experience into the final interpretation
	ii. Consensus building towards final results
	iii. Assignment of manuscript(s) responsibility

Refining the original published studies

Before the first face-to-face meeting, participants select (Circle A, #1 under refining in Figure 1) original published peer-reviewed manuscripts or research reports from their program of research that are considered to be relevant to the original broad aim. The original broad aim was: ‘To perform a synthesis of comparable studies to better understand the impact of primary healthcare reform on the organization, routines and relationships within primary care practices in different health care settings’.

The next step happens at the first face-to-face meeting and requires finding a more specific focus from the original broad research aim (#2 under refining in Figure 1). This involves brainstorming, pile sorting and concept mapping. For example, we refocused our aim on the more specific question: ‘In what way do primary care reforms influence the development of teamwork in primary care practices’, because the investigators’ unpublished experiences suggested teamwork was a critical aspect of reform even though it had not been a focus in 7 of 12 studies or many of the publications. This step distinguishes CRDA from other current meta-synthesis approaches which all require that the focused question is known before the synthesis and then limits the included studies to those published and explicitly addressing the focused question. CRDA brings both experiential tacit knowledge and unpublished data into the process of question focus and material available for the meta-synthesis.

The refining process involves iteratively reviewing and selecting manuscripts or reports to be included in the re-focused synthesis. A crucial aspect of our selection process was the participants’ common interest in the development of teamwork in reforming primary care service delivery (the object of the focused synthesis aim) and their diversity of perspectives. The presence of published material and accessibility of non-published data for reanalysis, as well as availability of information about the context in which the study was conducted, are also criteria for selection. This selection process resulted in the inclusion of published and unpublished material from 12 mixed-methods studies, 3 from Australia, 5 from Canada and 4 from the USA, where one or more models of primary care reform were studied (Table 1 describing the context, objectives and main components of each of the included studies).

The next step is to develop an extracting matrix to extract data from the published studies (#3 under refining in Figure 1). For example, our initial matrix was based on the realist evaluation framework which allowed us to extract information pertaining to context, mechanisms and outcomes from each selected manuscript.

Organizing broader study materials

The broader study materials (Circle B in Figure 1) consist of additional information and insights that are not objects of prior selected manuscripts. These materials include additional relevant results or data from each original study, such as other published findings from the study, unpublished analysed data, and even unanalysed data. For example, in one of the studies from the United States there were more than 20 published manuscripts beyond those selected, and substantial qualitative data referring to teamwork that was not published because it was not part of the primary intent of the study. These materials need to be classified according to level of evidence quality (#4 under organizing in Figure 1). In our case, we discerned three levels: peer-reviewed publications (Level A), investigator re-analysis of their data for teamwork (Level B) and interviews of project members about insights on teamwork (Level C). With the additional data there is a high likelihood that new matrices will need to be created for re-extracting data for ongoing analysis (# 5 and 6 under organizing in Figure 1). The iterative classifying, re-extracting and analysing tasks represent an organizing procedure using analytic matrices. This enables the team to take material that was initially very heterogeneous with respect to purpose, context, and methods and organize it to be more and more homogeneous with regards to the focused synthesis aim (e.g. primary care teamwork). This process involves going back to the original data to fill the gaps in analyses, provide new insights, triangulate and explain. This also results in expanding and refining the analytic matrices, including reaching a new level of understanding of published materials and re-analysing of data from included manuscripts.

Interpreting collaborative knowledge

Collaborative knowledge (Circle C in Figure 1) in the CRDA methodology refers to relevant knowledge which CRDA participants gain from their collaborations with other research teams. Collaborative knowledge represents what the senior investigators from other research teams would have brought to the CRDA process if they had been a part of it. For example two of the CRDA participants had worked collaboratively on projects that were led by an investigator who was not part of the Catalyst Team, who was able to provide additional insights from studies not included in our process (41–43).

This process begins by breaking the analysis into small groups and using conference call meetings. In the context of the small groups, the new collaborative knowledge is absorbed and entered into the matrices. The small groups continue the work of re-interpreting around expanded matrices on context, mechanisms and findings. In this iterative process of reviewing current analyses, absorbing newly introduced collaborative knowledge and reinterpreting and refining analytic matrices, the team reaches consensus on definitions and fully integrates all the information into the matrices.

Integrating experiential reflections

Experiential reflections (Circle D in Figure 1) relate to each participant’s accumulated lived experience and knowledge, both as an investigator and as a person living in their particular context, including participants’ initial interpretive summaries of the findings up until this point in the process. These individual perspectives are publicly shared in the interpretive space for group re-interpreting, reflecting and integrating into the final synthesis (#8, #9 and #10 under integrating in Figure 1). Reflecting and integrating activities involve questioning the concordance of current findings with the new shared experiential reflections. This phase allows one to situate the insights emerging from the CRDA into broader contexts before final integration. For example we began to understand that physician autonomy and dominance was a huge barrier to team formation and potentially overrode any external factors being proposed and that nobody was acknowledging its importance and making recommendations for changing this dynamic. This finding about physician autonomy was not apparent from the published studies, but became obvious while re-interpreting the studies with the tacit knowledge gleaned from the researcher’s experiences and reanalysed unpublished data.

Results

Our iterative approach in developing the CRDA using facilitated face-to-face meetings conducted over several days was found to be invaluable in generating productive and creative conversations and opportunities to delve deep below the surface of published research (Table 3). The CRDA process creates an environment where interactions among diverse participants lead to the emergence of ideas that transcended the original questions of the investigators and the studies they contributed to the process. Established investigators certainly bring more tacit knowledge and lived experience to the collaborative table; however, we found ‘expertise’ often resulted in considerable jostling among participants wanting to be heard. As a result, having a facilitator proved to be invaluable for the face-to-face meetings.

Table 3.

Catalyst team members experience with each phase of the Collaborative Reflexive Deliberative Approach

CRDA phase	Catalyst team experience with CRDA phase
Refining original published studies (Circle A in Figure 1)	In August 2010, we converged on the picturesque colonial era town of Lambertville, New Jersey on the Delaware River, each of us equipped with publications and reports generated through large-scale projects in which we had participated from our particular context. We each optimistically hoped our own work would contribute to a comprehensive understanding of primary health care reform, and in retrospect, had the simplistic idea that we would ‘compare and contrast’ and ‘cut and paste’ our results into common themes that would generate a comprehensive cross-contextual approach to primary healthcare reform. However, it soon became clear that we needed to understand the real story behind each of the ‘published stories’ in order to transcend contexts. To find the ‘gems’ that cut across stories, we were compelled to capture core concepts from each study on yellow cards and placed these on a huge storyboard (the floor). Everyone stood around contemplating and some eventually began ‘pile sorting’, physically moving cards back and forth—all the time negotiating meaning and intent, much like a distant image coming in and out of focus until finally commonalities of perspective and meaning emerged. Diversity of players was important to this process, but so was having an outside facilitator to keep the discussions moving with some coherence. While some took a big picture perspective and pulled diverse threads together, others listened and extracted concrete commonalities. It was amazingly generative, frustrating, stimulating, productive and fun. These were not words we had commonly used to describe our research, but it was definitely an experience that renewed our faith in the power of research to contribute to change.
Organizing broader study material (Circle B in Figure 1)	We left Lambertville with enthusiasm and a lot of great ideas, but also knowing we would need another face-to- face meeting before our next major gathering. Fortunately, most of our collaborative team was planning to attend an upcoming NAPCRG conference, so we reserved a conference room for the afternoon of the final day of the conference. Early in this 5 h intense sprint one of the participants led off with, ‘I went back this morning to the Pawson realist synthesis paper and it’s not clear what the difference is between mechanism and context’. That’s all it took to get us on a prolonged rant on definitions and strategies for completing matrices. In fact, a whole host of challenges to completing the matrices had emerged over the previous 2 months as investigators attempted to fill in cells from their studies. This meeting became critical for hashing out differences in definitions and coming up with clearer and shared understanding for what went into each cell. Having the same facilitator from our initial retreat was critical; although this time she was present only via Skype and clearly had challenges in keeping things on track. By the end of the meeting, we fully realized that for our next retreat to be successful, we would need to have a smaller group really focus on completing the matrices.
Interpreting collaborative knowledge (Circle C in Figure 1)	Coming off our brief second face-to-face meeting at NAPCRG, a smaller analysis group was formed and met monthly to prepare for what was to be our defining meeting at Sorrento, Victoria, Australia in February 2011. The initial meeting of this smaller group began by summarizing decisions made by the larger group at NAPCRG and expanding on the analysis from that meeting. We paid particular attention to changes in the research questions and clarification of how we were using the three matrices (context, mechanisms and findings), being explicit about how we defined the rows and columns that populated each matrix. We quickly acknowledged the difficulty of ‘analysing from a distance’. Our second meeting focused specifically on the mechanisms matrix. There was some difficulty for the group in coming up with a shared definition of ‘mechanisms’ for this study and it was sometimes conflated with ‘findings’. One member said we needed to be ‘nose-to-nose’ to clarify the meanings we were making of the analytic matrix and data analysis. We decided to focus on the findings matrix and use this, along with the context matrix, to prepare for the next face-to-face meeting and discussed how we could make our short time together most productive. It was decided that one member would start a cross-case analysis, present these interim findings to the larger group at the next face-to-face meeting and that would springboard the discussion into further analysis. As the face-to-face Sorrento retreat neared, the analysis group reflected on the logistics of organizing the meeting, all the while keeping in mind that the Sorrento retreat needed to develop a conceptual framework, plan the writing of publications, and identify options for future studies to enable the group to keep working together and further develop our findings.
Integrating experiential reflections (Circle D in Figure 1)	After 8 months of really getting to know each other by iterating ideas and painstakingly completing data matrices, the collaborative Catalyst team reconvened for its final multi-day retreat at the Hotel Sorrento overlooking Port Phillips Bay in Victoria, Australia. We had done a lot of work to this point, but we were far from agreement and really needed this 2½ day retreat to solidify our thinking. With flip-chart stands in each corner for jotting ideas, a large white board, and a computer projecting onto a large screen for taking notes, we were ready for what we hoped was the final push. To get us started, Will Miller summarized five initial conditions and eleven findings that came from the summary. Almost immediately there was a lot of bantering back and forth, but there appeared to be partial agreement on most of these. By lunch of the first full day, we had a couple of model diagrams on the white board and by the end of the day we had made a lot of progress on a general outline and some key propositions that would drive a manuscript summarizing the deliberative synthesis. Nevertheless, we were still struggling with the terminology and found words were easily muddled. At one point, Mark suggested that changing concepts to verbs might help, using words like connect, communicate, coordinate, cooperate, collaborate and integrate. The group, especially the physicians, was really struggling with the discovery that physician autonomy and dominance was potentially the most potent barrier to team formation. Thankfully the work was interspersed with walks along the shoreline, regular breaks and congenial chats over meals.

Throughout the CRDA process, we recognized that the methodology managed two valuable tensions simultaneously. One of these was finding sufficient commonality while assuring adequate diversity. That is, it is necessary to pool similar studies while conducting an analysis, classification that takes advantage of the diversity of disciplines, methods, time and scope of both participants and studies. The second tension was about wanting to universalize findings (i.e. generate statements that apply across studies) while simultaneously respecting the distinct characteristics of each jurisdiction’s context.

CRDA endeavours to generate knowledge about an object or focus (in our case, teamwork in primary care). Obviously, despite variations in how different studies operationalized the main object under study, there were commonalities to be found. For example all of the studies that had been conducted by participants in our collaboration were about primary care practice in a context of health systems reform and included data of sufficient quality and quantity concerning inter-disciplinary teamwork. Commonality of object is important to make sure that pooled studies and experiential knowledge relate to the same subject or intervention.

Having identified a common object for analysis, we needed adequate diversity in disciplinary perspectives, methods, time and scope to be able to assure the validity and transferability of our core findings. Some studies from this pool of participants took the angle of understanding the interdisciplinary care and clinical routines, while others looked at organizational models and quality improvement initiatives. There was also diversity in underlying models for understanding organizational change; several were grounded in the use of complexity science while others drew upon organizational theories and more linear models. These complementary perspectives on a same object (teamwork in primary care practice) provided the basis on which to pool these studies and extract new emergent knowledge from each, not directly related to the original perspective taken.

Discussion

In operationalizing the methodology, we discovered that it had the potential to foster triple loop learning and accelerate breakthroughs. The organizational literature differentiates three types of learning: single, double and triple loop learning. Single loop learning refers to incremental improvement whereby error is detected and corrected. It answers the question, ‘Are we doing things right?’ Most performance improvement approaches and process evaluations are about single loop learning, as are most hypothesis-testing studies. Double loop learning answers the question, ‘Are we doing the right things?’ The result is a reshaping of ways of thinking and a change in underlying assumptions (44). Developmental and utilization-focused evaluations and the meta-synthesis approaches seek double loop learning, informing and changing the underlying conceptual framework. Triple loop learning, or transformational learning, represents changing our understanding of ourselves and our context, a paradigm shift, and creating a collective mindfulness across a system (45). Triple loop learning is often what is needed when dealing with problems where a field of research appears stuck, and there is disagreement both in the science and in the public arena which is the case for teamwork in primary health care delivery (46). CRDA methodology invites disagreement and uncovered knowledge into a common space and evokes new understandings through triple loop learning.

Three major findings emerged from our CRDA study. The first concerned the identification of four dimensions of collaboration within teams and that the fourth dimension, functional collaboration, was the most neglected, yet essential for long-term success (47). The functional dimension specifically involves the definitions of roles and responsibilities aimed at coordinating the team’s activities. One of the reasons this dimension of collaboration is so difficult relates to our second finding that teamwork success varied more within a jurisdiction than between them and that inter-professional dynamics were the key determinant even more powerful than external policy (48). The third finding, regarding the levers for enhancing teamwork, found that physician dominance, lack of well-developed corporate governance systems and physician-aligned fee-for-service payment were the most significant (49, pending this issue of Family Practice). Current conceptual frameworks for developing teamwork and other interventions in primary care nearly all make minimal or no reference to this factor of physician autonomy and dominance (50–53). The finding concerning physician autonomy and dominance represents double loop learning, but for the physician members of our team, these results were difficult to accept and required changes in self-understanding demonstrating triple loop learning.

There are a number of limitations and caveats with CRDA. Face-to-face meetings are expensive and require commitment and time. Having a facilitator appears necessary, and even with facilitation, the group may not form as an effective team, and it may be necessary to adjust membership over time. As we operationalized this process, there was the potential for selection bias and insufficient diversity to keep the group challenging existing perspectives. We also had the bias of not including early career investigators or people with whom we didn’t think we could effectively collaborate. In our process, we included a policy maker to help keep the discussions focused and policy relevant, but this remained limited since the single policy maker represented only one jurisdiction. In the future, it may be better to include pairs of academic investigators and policy makers from each jurisdiction or at least one policy maker from each context. We also did not include patients in our team, which undoubtedly would have changed the conversations.

Because CRDA requires a large commitment of investigator time, strong leadership, considerable coordination and resources for convening facilitated face-to-face retreats, it should be considered primarily for important questions in which there is broad interest, especially those that seem intractable. For CRDA to work, careful attention needs to be paid to ensuring sufficient diversity among participants while also selecting people who can collaborate well and effectively using democratic group processes. Careful attention also needs to be paid to ensuring that there are enough diverse studies to assure contextual variation. Once participants and studies are identified, it is necessary to create a space where diverse people feel psychologically safe.

Conclusions

Meta-analyses and meta-syntheses represent increasingly popular strategies for blending messages from multiple sources of published literature, but neither approach involves the original researchers synthesizing or uncovering tacit and unpublished knowledge. CRDA expands on these strategies by focusing on carefully selected investigators and their programs of research and explicitly and systematically combining the published literature, unpublished insights and investigator experiences. This methodology provides a richer description of what lies between the lines of published studies and provides a formal process for investigators to be more reflective about the research process.

Declaration

Ethical approval: the Ottawa Hospital Research Ethics Board (#2010-270-01H).

Funding: the Canadian Institutes of Health Research Catalyst (PCH—103625).

Conflict of interest: none.