Creating and Supporting a Mixed Methods Health Services Research Team

Abstract

Objective. To use the experience from a health services research evaluation to provide guidance in team development for mixed methods research.

Methods. The Research Initiative Valuing Eldercare (THRIVE) team was organized by the Robert Wood Johnson Foundation to evaluate The Green House nursing home culture change program. This article describes the development of the research team and provides insights into how funders might engage with mixed methods research teams to maximize the value of the team.

Results. Like many mixed methods collaborations, the THRIVE team consisted of researchers from diverse disciplines, embracing diverse methodologies, and operating under a framework of nonhierarchical, shared leadership that required new collaborations, engagement, and commitment in the context of finite resources. Strategies to overcome these potential obstacles and achieve success included implementation of a Coordinating Center, dedicated time for planning and collaborating across researchers and methodologies, funded support for in-person meetings, and creative optimization of resources.

Conclusions. Challenges are inevitably present in the formation and operation of effective mixed methods research teams. However, funders and research teams can implement strategies to promote success.

Mixed Methods in Health Services Research

Despite representing a minority of published health services studies (Wisdom et al. 2012), mixed methods health services research has increased significantly over the past decade (O'Cathain, Nicholl, and Murphy 2009; Curry et al. 2012; Zhang and Creswell 2013). Methodologists and researchers increasingly suggest that mixed methods designs are particularly important for health services research as it is, by nature, highly complex (Brannen 1992; Castro et al. 2010; Creswell and Plano Clark 2011), and so must capitalize on the strengths of both qualitative and quantitative approaches (Creswell et al. 2011; Wisdom et al. 2012). Adding to prior evidence that mixing methods confirms the accuracy of findings, accommodates greater comprehensiveness, provides insight into the processes linked to outcomes, and is useful when investigating “hard to measure constructs” (Creswell et al. 2011), widespread agreement now exists that a combination of methodologies is needed to understand the circumstances under which change (i.e., interventions) works. Considering the current emphasis on translational research and the importance of practical application to stakeholders, there is recognized need to contextualize findings and understand the reason for and mechanism of change (Alexander and Hearld 2012; Zhang and Creswell 2013). Mixed methods designs are seen as vital to achieving this goal; however, the paucity of guidance in mixed methods procedures presents a notable challenge to their full realization (Greene, Benjamin, and Goodyear 2001; Bazeley 2009; O'Cathain 2009; Bergman 2012; Wisdom et al. 2012; Zhang and Creswell 2013).

A fundamental component of mixed methods research is the research team. Although some researchers (Hall and Howard 2008) suggest that a single researcher can conduct mixed methods research, most recognize how unlikely it is for a single researcher to possess sufficient methodological expertise to carry out a rigorous mixed methods study (Rallis and Rossman 2003; Johnson, Onwuegbuzie, and Turner 2007; Teddlie and Tashakkori 2009; Sharp et al. 2012; Torrance 2012; Youngs and Piggot-Irvine 2012). Acknowledging the importance of team development and management, Creswell et al. (2011) provide a beginning outline of the various types of research teams that might be used for mixed methods research, the components of effective research teams, and the steps involved in forming, leading, and training research teams. Sharp et al. (2012) acknowledge the need to consider relationships among and between team members as well as the expertise they might contribute. How this might be done, and to what end, however, has not been elaborated. In addition, a challenge reported by several researchers is the emergence of power differentials and “methodological disrespect” among team members, apparently emanating from perceptions of methodological superiority (O'Cathain, Nicholl, and Murphy 2009). Bergman (2012) terms this challenge the “uneasy compromise” (p. 273) of mixed methods teams. Such discussions suggest that methodological expertise in itself may be insufficient to develop, maintain, or maximize the effectiveness of a mixed methods team.

With a few notable exceptions (Creswell et al. 2011; Curry et al. 2012), the silence on how to develop and implement an effective mixed methods research team is striking. Although published accounts of mixed methods research often acknowledge the significance of the team, they rarely provide insights into how that expertise is used in the research process, how the team functions, what challenges are encountered by the team, and how those challenges can be overcome (Sharp et al. 2012).

Several research reports and methodological papers have suggested that skilled team leadership is vital for a successful mixed methods research team but have not offered strategies in which leaders might engage to develop and maintain the team (Jang et al. 2008; Creswell et al. 2011). Others have indicated that leadership can be shared among group members, or that team members can step forward to take the lead when their expertise is required. However, we found no descriptions of how the right moment to assume leadership is determined, or how leadership changes. Furthermore, this suggestion assumes that a given individual possesses the skills to lead a diverse team.

Youngs and Piggot-Irvine (2012) move a bit closer to the question of team leadership in mixed methods teams by distinguishing between “team-centric” and “leader-centric” leadership, noting that the project leader plays a key role, particularly in the early stages, enabling the team to develop shared leadership. Although they note that the project leader facilitates the distribution of shared leadership across the team, they offer no details about how this may be achieved. A few authors have suggested that at least some members on a mixed methods team must have prior experience conducting mixed methods studies (Arnault and Fetters 2012). However, just what skills or expertise this past experience provides in relation with maximizing the effectiveness of the team, and what can be done when none of the team members has such expertise, was not addressed. Curry et al. (2012) provide a rare in-depth discussion of mixed methods team dynamics. They explore how other group affiliations influence the dynamics and effectiveness of the mixed methods research team and recommend several useful principles for managing resulting tensions.

Building on the work of Curry et al. (2012), this article describes how a mixed methods research team “The Research Initiative Valuing Eldercare (THRIVE)” experienced and addressed challenges to team development, including strategies to develop and organize the THRIVE team and incorporate the funding agency as an integral part of that team, and the impact of this integrated team approach on study design and conduct. The article includes strategies that are potentially generalizable and may be useful to other mixed methods efforts.

Importantly, the THRIVE team developed and worked within a “fully integrated” mixed methods research model, wherein all activities—including resource allocation and sharing, project and measure design, data collection, analysis, interpretation, dissemination, and so on—required collaboration by and active engagement of every member of the research team. This fully integrated model contrasts sharply with other, less integrated mixed methods research models, whereby, for example, one researcher completes the quantitative analysis for a project while another completes the qualitative analysis, and then these researchers come back together to author a manuscript. For the remainder of this article, the term “mixed methods research” refers to the type of fully integrated mixed methods research that the THRIVE project exemplifies.

The THRIVE Initiative: An Evaluation Using Mixed Methods Research

In 2010, the Robert Wood Johnson Foundation (RWJF) convened the THRIVE team to determine the impact of The Green House Project's model of nursing home care. The Green House Project represents an emerging model of nursing home care where small houses, centered around an open kitchen and central hearth area, are home to 8–12 residents who have private rooms and baths, and where care is provided by a consistent, self-directed team of caregivers who are responsible for maintaining daily home operations, including tasks to support dining and housekeeping. In this model, licensed nurses attend to resident's medical needs but are otherwise viewed as “equals” to the unlicensed caregivers. This model contrasts sharply with more “traditional” nursing homes, typified by large numbers of residents (national average = 108; National Center for Health Statistics 2006) who often reside in shared rooms, in buildings with long corridors and nurse's stations, which are more similar in style and atmosphere to a hospital than a home. Traditional staffing patterns generally are hierarchical, with nursing and other clinical staff supervising caregivers who work in relatively specialized roles. Despite anecdotal reports of better care and outcomes in Green House settings, the scant research to support these claims is inconclusive. Indeed, a 2010 review concluded that although some of the components adopted by the Green House model (e.g., private rooms and bathrooms, open kitchen, communal dining) previously had been associated with better outcomes, the evidence to support other components adopted by the model (e.g., consistent assignment, small size) was either mixed or inconclusive (Zimmerman and Cohen, 2010).

Despite the modest and potentially mixed evidence, the Green House model has attracted attention from consumers and policy makers, and has grown significantly over the past decade, from only a handful of homes in operation to over 100 in 2012, with many more in the planning phases. Given this, RWJF's goal was an evaluation to (1) provide state and federal policy makers comprehensive evidence pertaining to the impact of The Green House model compared with other models of nursing home care and (2) inform long-term care organizations, investors, funders, and policy makers seeking to replicate and sustain The Green House model.

Supported by RWJF, the THRIVE team collaboratively identified six primary research aims that were addressed through interrelated THRIVE projects funded under separate grants. The brief titles, descriptions, and methodologies used to address each aim are provided in Table 1. Figure 1 provides a graphic example of the interrelated nature of these aims and the projects to address them. Figure 1 also shows the Coordinating Center, comprised of members of the project teams as well as the RWJF project officer, which was created to support, coordinate, and maximize the integration of the various projects.

Table 1.

Design and Purpose of THRIVE Projects

Focus of Study (Short title)	Team Code	Methodology (data source)	Study Aim(s)
Culture change	A	Quantitative (online questionnaire)	To study “exemplar” NHs to describe the similarities and differences in components of culture change (i.e., the term describing changes in nursing home care that seek to achieve processes and outcomes that are better aligned with the values and desires of the person receiving care) adopted by these homes
Implementation	B	Qualitative (inductive study, open ended to structured interviews)	To identify and describe core elements of the Green House model, from the viewpoints of staff of The Green House organization and individual Green House homes, and to examine how the latter implement these initially and over time
Clinical outcomes	B	Primarily quantitative (mixed methods questionnaire, structured interview, NH records, structured observation)	To identify structures and processes of care that relate to resident outcomes in traditional, culture change, and Green House NHs, and to determine how structures and processes vary across setting types
Workforce	C	Quantitative (NH records)	To describe and compare the characteristics of staff working in traditional, culture change, and Green House NHs, and to determine and compare the turnover of these staff, and the setting and staff characteristics that relate to turnover
Outcome- related processes	C	Qualitative (semistructured interview, content analysis)	To identify, describe, and assess the sustainability of, and clinical care practices and processes associated with, better resident outcomes
Costs/quality	D	Quantitative (secondary records)	To determine the characteristics of NHs that later go on to adopt the Green House model, and to evaluate the impact of Green House on the cost and quality of care

NH, nursing home; THRIVE, the research initiative valuing eldercare.

Figure 1.

Interrelationships and Interdependencies of the Research Initiative Valuing Eldercare Team

Creating and Sustaining a Mixed Methods Research Team

Mixed methods research teams generally include researchers from diverse disciplines who embrace diverse methodologies. They often involve new collaborations and require significant levels of leadership, engagement, commitment, and sharing of resources. Each of these components may present challenges; the experiences from the THRIVE team in overcoming these potential challenges are discussed below and summarized in Table 2.

Table 2.

Strategies to Address the Theoretical and Practical Challenges of Mixed Methods Research

Theoretical (Literature)		Practical (THRIVE)
Component	Challenge	Challenge	Solution	Benefits	Considerations
Diverse disciplines	Different locations and policies Different knowledge base Different language Different goals Different target audiences	Team comprises nurses, economists, health services researchers, social workers, statisticians, advocates, and others Team members use different language Goal for some is to publish definitive results in research journals; for others, goal is to use preliminary findings to quickly inform policy and practice.	Ongoing group meetings with dedicated time for each project and person Coordinating Center has strong knowledge base of each person's background and each project's design Continued discussion of goals and audience, and some compromise Written plan for interim and final products and guide for sharing with diverse stakeholders	Colearning about other disciplines Broader perspective about research impact More diverse, far-ranging audience for findings	Significant time should be devoted to develop sharing and publication policies Compromise is essential
Diverse methodologies	Risk of methodological marginalization Lack of understanding (of process, results, interpretation)	Team comprises quantitative and qualitative researchers Differing views on evidence required for interpretations of findings Quantitative measures (and thus data collection) develop slowly, whereas qualitative work dependent on the former develop more quickly	Funder organized a team of leading, senior researchers Time devoted to understanding each project's method and criteria for rigor Researchers worked together to develop and revise timelines and sequencing acceptable to all	Each methodology contributes different knowledge Projects can inform and refine one another	Team members must be open to new approaches, and without “ego” Flexibility is key
Nonhierarchical leadership	Potentially diffuse organization No clear leader	Team of diverse “equal” researchers with no leader No individual with dispute resolution authority Funder a member of research team, but still need a “point person” for communication and accountability	Coordinating Center created to identify unmet needs, arrange meetings, attend to logistics, facilitate decision making, liaison with funding agency and outsiders, and support development of team cohesion	Researchers able to focus on the research, with minimal time required for logistics All ideas equally and democratically considered and decided Coordinating Center handles tasks that benefit all team members	All must acknowledge coordinating position as that of democratic facilitator, not dictatorial leader Group should honestly consider characteristics needed in a coordinator to ensure appropriate fit Coordinating needs will evolve throughout project
New collaborations	No pre-established trust or working relationships	Few team members have ever met, and have not worked together before	Coordinating Center arranged weekly phone meetings Annual in-person meetings that include social (i.e., non-work) time	Team members genuinely like and respect one another New professional relationships spawned; many have since joined together voluntarily on other research efforts	Weekly meeting agendas should include ample time for team discussions to foster respect, trust, and better understanding Team-building is time-consuming (and thus costly) Budgets should include funds for in-person meetings
Team engagement and commitment	Independent projects easier to coordinate Time consuming and sometimes inefficient	THRIVE comprises topically and methodologically distinct projects Team members have less interest in some topics and methods Long-term project poses challenge to continued engagement	Regular meetings Funder present at meetings Agendas include mix of project-specific items, as well as bigger picture ones Ongoing consideration of how projects are interrelated, so what may have seemed irrelevant to one researcher becomes relevant Coordinating Center does outreach to spark outside interest and collaboration	Funder stayed involved in decisions and progress Incentives for team to stay engaged Funder provided material and personnel resources to help Synergies and opportunities across projects identified Outsiders provide THRIVE team with other research opportunities	Maintaining an engaged and committed team is crucial Regular meetings are essential, even though the time spent is rarely fully funded
Optimal use of resources	Mixed methods research is difficult to coordinate Mixed methods research has potential to overburden subjects Costly	THRIVE projects include many of the same research sites and subjects Data redundancies No funding to support ongoing collaborations with other researchers	Site communications were centralized through Coordinating Center Team reviewed all planned sites, subjects, and data requests beforehand to remove duplication Each researcher collected some data for another researcher	Research sites and subjects are not confused or burdened—data collection appears seamless Permitting researchers to collect other researchers data increases everyone's investment Costs are minimized by not duplicating efforts	Data use agreements must be put in place beforehand Each researcher (via his/her institution) will have different data sharing rules that must be attended to All researchers must be comfortable with how data are collected

THRIVE, the research initiative valuing eldercare.

Nonhierarchical Leadership

Research coordination is a complex, time-consuming, and, at times, inefficient process. Some have suggested that leadership can be shared across team members in mixed methods groups, and that members can lead individual efforts on an “expertise-as-needed” basis. However, such expectations are somewhat naive, ignoring the many leadership skills that require the ability to coordinate, summarize, and set and monitor goals, and the risk that diffusion of responsibility creates. Anticipating this challenge, the RWJF established a Coordinating Center tasked with focusing on inter-relationships among studies; identifying and meeting logistical and project-related needs; arranging and preparing for meetings; facilitating decision making; and liaising with outsiders and the funding agency (RWJF). Two researchers with extensive knowledge of the substantive area, experience managing large grants, appreciation for a range of methodologies, and a demonstrated ability to collaborate and negotiate, were selected to work alongside the project officer in a Coordinating Center role, assisted by an individual skilled in overseeing programs of research.

Initial responsibilities of the Coordinating Center included identifying and inviting team member participation, prioritizing the funding agency's research interests, negotiating budgets, identifying synergies across researchers and projects, developing an initial conceptual framework for cross-researcher integration, and building a trusting, well-functioning team. Many of these initial activities were conducted in isolation of the team, but in collaboration with the project officer. This strategy was useful in maintaining integration while ensuring alignment of the project with the funder's initial goals. Throughout all phases of the research, the Coordinating Center's attention to logistical tasks permitted the other team members to focus solely on the research. Importantly, the Coordinating Center functioned as a “democratic facilitator,” not as a “dictatorial leader,” during both the initial and later phases. This style was especially important given that the team members comprising this entity were THRIVE research team members.

Another central feature of the Coordinating Center was that it assumed responsibility for specific research tasks that benefited all individual projects, such as developing a framework for the studies; establishing and maintaining contact with sites; preparing presentations for national meetings; creating, housing, and maintaining common databases; and identifying and promoting opportunities for dissemination—including related to this manuscript. In this way, the Coordinating Center is responsible for dissemination above and beyond that related to any one project or methodology.

Diverse Disciplines

To achieve the broad perspective that best serves a mixed methods approach, it is generally desirable for the team to comprise diverse disciplines. Such a composition can pose challenges, however, as the very definition of “discipline” suggests each has its own rules and expectations as well as invisible group affiliations that influence priorities and approaches (Curry et al. 2012). Disciplines differ in their training, knowledge base, language, goals, and constituent audiences.

Members of the highly diverse THRIVE team were selected by the funder to include experts with an established body of research relevant to The Green House model, a unique strategy from the outset. The researchers represented multiple locales (seven states in three time zones), disciplines (economics, social work, nursing, statistics, health services, and health informatics), and settings (universities, contract research organizations, advocacy groups). Although team members varied widely in disciplinary background, they shared a common interest in nursing home care and outcomes.

Despite this shared interest and goal, the team experienced initial challenges in their different use of “language” ranging from the term used to describe individuals living in the homes under study (patients, residents, or elders) to “impact analysis,” which was viewed quite differently by the economists and the clinicians on the team. Such differences were numerous, and not always immediately obvious, posing a risk for team division, marginalization, and misunderstanding.

To avoid potential conflict, the Coordinating Center created a collaborative listserv for sharing emails, and also established weekly telephone meetings. Although the weekly meetings included substantive research agenda items, they also included time for the team to foster friendliness, trust, and understanding of one another's work, and to discuss accountabilities to each other's study. In addition, the funding agency not only supported—but required—that all team members budget time for these meetings, as well as travel to attend in-person meetings. These in-person meetings were invaluable in establishing trusting partnerships.

Through these meetings, differences and misunderstandings were uncovered, discussed, and resolved. Importantly, and informed by prior reading of each team member's previous work, the Coordinating Center proactively identified areas of misunderstanding and potential conflict, bringing these issues to the attention of the team for discussion and resolution.

Another challenge for the team was the difference in their primary constituent audiences and, consequentially, in group members' priorities. For example, the academic researchers planned to publish findings in research journals. Definitive findings, ready for peer-review, require that all data collection, data cleaning, and analyses are completed, reviewed, and carefully crafted into a manuscript—a process that can take a year or more to complete. This goal and the associated timeline was at odds with those of other team members, whose goal was to quickly provide select stakeholders (e.g., policy makers, providers) with preliminary findings to use for quality improvement and policy development.

Recognizing the importance of each and the funder's desire for both, the THRIVE team devoted numerous meetings to identifying early products that would be useful for policy and practice improvement as well as crafting policies to guide the content and timeline for sharing information. This strategy was the outcome of multiple conversations on how to be responsive to different constituent groups while recognizing all members' priorities. The Coordinating Center, and particularly the funder, was clear from the inception that providing products useful for real-time quality improvement was an important goal of the evaluation. This negotiation was one instance in which the presence of the funding agency's project officer was helpful in that it implicitly promoted compromise.

Diverse Methodologies

Just as the THRIVE researchers hailed from distinct organizations and disciplines, they also differed in epistemology. Although paradigm issues are increasingly seen as irrelevant (Bazeley 2009; Bergman 2010), they were experienced by the THRIVE team as significant and were particularly apparent when producing unified messages for external audiences. This difference was most notable in relation to causality. Although all agreed that qualitative methods allow for deduction and induction, and that quantitative associations suggest potential causal relationships, team members differed in their interpretations of data (findings) generated from qualitative methodologies and in what was required to interpret associations as causal. These differences have been identified in other teams as well (Curry, Nembhard, and Bradley 2009; Wisdom et al. 2012). Furthermore, although not observed among the THRIVE team members, published reports have described the marginalization of qualitative researchers on mixed methods teams, particularly over the credibility of interpretations from qualitative studies (Curry et al. 2012; Denzin 2012).

To overcome these challenges of interpretation, THRIVE team members spent time early in the project reading selected articles written by other team members. This activity was followed by discussion of how the methodology used by each could contribute to the whole; what could be learned from each study type; what claims could be made; and how the different methodologies fit into the overall project. These discussions increased understanding of and respect for the differing contributions of individual team members. In addition, as part of their written policies, the THRIVE team incorporated a “sign-off” process for newly generated results and messages, to ensure that all team members were comfortable with the interpretations and conclusions being reached and disseminated by others.

Another example of a challenge arising from methodological diversity was the differing timelines of interrelated projects. Adhering to a true mixed methods design, the focus of the Outcome Related Processes study was to be determined by the results of the Clinical Outcomes study. However, due to the time associated with preparing quantitative measures, the latter study was not conducted until months after the former was intended to be conceived, rendering the initial intent impossible. By reviewing timelines and identifying interdependencies early on, the THRIVE team was able to adjust the timeline of the Outcome Related Processes study accordingly. Clearly, flexibility regarding the timeline and focus of projects is key, if they are to inform and refine one another as per the intent of mixed methods research. A high level of engagement from all team members—including the project officer—facilitated this process and led to a design acceptable to all, preventing any team members from feeling marginalized, and permitting the funding agency input if the final design yielded an unacceptably long timeline.

New Collaborations

The THRIVE initiative involved the establishment of several new collaborations. All matters of organization, methodological preference, and discipline aside, new collaborations always present challenges, especially when they require dependence on other team members' knowledge and expertise. Few of the THRIVE team members had met previously, and only two had collaborated previously. The email listserv communications, telephone, and in-person meetings established by the Coordinating Center were essential in forming a cohesive, collaborative team.

Beyond prospective efforts to support these new collaborations, in retrospect, we realize the THRIVE team benefited from the serendipity that all team members were highly interested and dedicated to the topic and methodology, had personalities amenable to new and uncertain collaborations, and were willing to both lead and follow partners. Each of these characteristics should be considered for future collaborations, as all are essential to team formation and functioning.

Team Engagement and Commitment

Maintaining engagement of team members and ongoing commitment to the overall project created a potential challenge. The geographic and methodological diversity, and the time required to work collaboratively, could easily have resulted in silos and a failure to integrate the separate projects.

Common to many mixed methods projects, THRIVE required an extended timeline, meaning that engagement and commitment needed not only to be created but also sustained over time. The THRIVE team achieved this goal with regular, weekly (and later, twice monthly) meetings, regardless of whether the agenda was pressing or there were other competing demands for time. The RWJF project officer attended all meetings, which served to both increase accountability and permit her to contribute to the decision-making process. This level of funding agency (RWJF) participation is not commonplace and signified both the project officer's personal interest, and RWJF's commitment to, the evaluation. The meeting agendas were crafted to include a combination of items specific to individual projects (so that all members remained involved in every project), as well as items of definite cross-cutting relevance.

Mixed methods research designs offer unexpected and sometimes unforeseen opportunities for processes and aims to inform one another. The likelihood of this serendipity occurring increases as more time is spent discussing each other's work. Such possibilities for new collaborations strengthened each team member's commitment to the projects and to each other. In addition to collaboration within the THRIVE team, the Coordinating Center worked to identify potential outside collaborations, which infused energy and opportunity into the team.

Optimal Use of Resources

As noted, mixed-methods research projects are difficult to coordinate at both an individual project and collaborative level, and are somewhat inefficient and costly. If not well coordinated and streamlined at the subject-level, these research projects carry the potential to overburden participating research sites and subjects. Given these challenges, mixed methods efforts must capitalize on all opportunities to streamline processes and recognize the benefits of the mixed-methods approach.

The THRIVE team chose to centralize all THRIVE-emanating communications into one member of the Coordinating Center. This individual was responsible for all initial communications with participating sites, even when this created a “middle-man.” From a site's perspective, however, it was easier to communicate with one person about multiple topics, rather than with multiple people about multiple topics. To accomplish this centralization, the THRIVE team reviewed each project's research protocol, including proposed sites, subjects, and data requests at the initiation of the project, so as to avoid duplication. In so doing, many synergies were discovered. For example, THRIVE discovered areas of common data needs, and by sharing data (as opposed to collecting it multiple times), researchers recognized time- and cost-savings. An added benefit of this process was that researchers became more engaged in the process of other projects out of investment in their own data.

Across projects, the group actively engaged in opportunities to optimize resources and expand their overall potential while minimizing effort and expense. Figure 1 highlights where researchers identified these synergies across projects in areas such as measurement development, data collection, and input into design. Each collaboration was unique and defined by the intersections of the projects. Validation of the Culture Change survey is one such example. The Culture Change project leader sought to validate the self-report survey but lacked necessary resources. Because members of the Clinical Outcomes team were already visiting sites that had responded to the survey, validation was a fairly simple “add-on” to onsite activities. In another case, the Clinical Outcomes team benefited from the Implementation team's prior on-site visits with some participating sites, and also by dividing some data collection responsibilities between the two teams.

Discussion

Appreciation of the benefits and increasing appropriateness of mixed methods health services research has outpaced the knowledge and skills required to optimally conduct these complex studies. Although numerous researchers have struggled with the challenges presented by mixed methods studies and lamented the lack of available protocols (Bazeley 2009; O'Cathain 2009; Bergman 2012; Greene, Benjamin, and Goodyear, 2001; Wisdom et al. 2012; Zhang and Creswell 2013), little practical guidance has been available. Silence on this matter has been striking particularly in the area of team development and functioning, an area that presents numerous, daunting, challenges but also tremendous opportunity. This article contributes to the literature by discussing some of the challenges and solutions encountered by one mixed methods research team (THRIVE), engaged in a complex, multifaceted health services research evaluation.

Increasingly, high-quality, practice-relevant health services research requires engaging teams of expert researchers representing a range of disciplines and methodological skills. Indeed, mixed methods studies are strengthened when research teams are composed of individuals from various disciplines (Sammons, 2010), ideally resulting in what Greene (2007) referred to as a “mixed methods way of thinking,” whereby not only methodologies but also epistemologies are mixed.

The THRIVE team's collaboration was strongly influenced by the disciplines involved in the project, and a different combination of disciplines would almost certainly have led to somewhat different decisions along the way, influencing the final product. The THRIVE experience highlights the importance of carefully considering the disciplines included, their specific contributions to the study, and the implications for how the study will proceed. For example, considering disciplinary conventions and interpretations of data within the context of each discipline will allow team members to anticipate at least some of the decision points that will likely arise, and allow team members to discuss these (and the implications for the overall study), in advance.

Being able to work effectively within a research team with such diversity, where many team members are accustomed to being most senior, and where trust and collaboration have not been previously established, requires new perspectives, skills, and investment of time and effort. Substantive and methodological expertise alone is insufficient for effective team operation. The THRIVE team experienced challenges in creating a new team, but these were overcome through nonhierarchical leadership, frequent team meetings in which open discussions were facilitated and promoted, collaborative policy creation, and, most important, flexibility and commitment.

Although the inherent commitment of the THRIVE team members was necessary and contributed to the success of the collaboration, commitment was deepened and sustained in a variety of ways. One not commonly seen in health services—and perhaps not in other—research was the sustained presence and involvement of the funding agency's project officer. Through regular participation in project meetings, the project officer clearly demonstrated the importance of the THRIVE projects to the funding agency. By staying involved and aware of the challenges the THRIVE team faced, the project officer better understood the difficult decisions that often must be made by research teams and, in some instances, was able to offer both material (e.g., funds) and other (e.g., personnel) resources to overcome these difficulties.

The funder's presence in team deliberations itself engendered collaboration and incentivized the participation and sustained engagement of the research team members. Rather than influence the research design or its implementation, the project officer instead assumed responsibility for reminding the research team of the funding agency's expectations of resource optimization and translatability; in so doing, the THRIVE team was held accountable for conducting a true mixed methods study that extended well beyond pooled data analyses that often constitute “mixed methods” designs.

Researchers can be both intrinsically motivated to conduct mixed methods studies by becoming more familiar with these designs, and extrinsically incentivized if funders require them to be so. The Patient Centered Outcomes Research Institute has had success in its well-known efforts to simultaneously educate researchers about and incentivize them to follow the community-based participatory research model of research, by requiring that all proposals implement components of this methodology. Other funding agencies may employ a similar mechanism to promote mixed-methods designs.

As this article illustrates, greater direction from and involvement by funding agencies may be effective, particularly in an era when funding agencies hope to obtain more for their limited budgets. Although such close involvement may not yet be standard practice, it does occur, especially in some contract work, such as that by the Agency for Health Care Research and Quality, the Assistant Secretary for Planning and Evaluation, and also NIH program projects. That said, and recognizing that funders may neither be able nor immediately willing to support this level of involvement in mixed methods research teams, another promising strategy could be for the centers, institutes, or individuals within universities charged with stimulating research to promote mixed methods research teams as part of their efforts. For example, the NIH Clinical and Translational Science Awards program, which already has a mission of interdisciplinary and engaged scholarship and is in 60 academic medical institutions across the country, could be a strong proponent to build such teams.

Finally, we do not believe that the absence of an involved funder precludes the possibility of an effective mixed-methods team. In the case of THRIVE, key roles of the funding agency included support for the Coordinating Center; promoting accountability and engagement among partners; and being attuned (and therefore more understanding) to the challenges and decision points that arise in research. Each of these key benefits can be created in teams that have strong, nonhierarchical leadership from a principal investigator or other leader who can assume the role that the funding agency assumed, allocating resources for coordination efforts (i.e., a “Coordinating Center”), and encouraging the team to create a collaborative system for thoughtfully navigating the decision-points and trade-offs that occur in research projects.

As funders and researchers alike increasingly recognize the scientific benefits of mixed methods designs in health services research, more new teams will encounter the challenges of team development and sustainability. The THRIVE team encountered and addressed these challenges, but recognizes its experience is only one case. That said, we believe that the foundational components of the THRIVE approach are generalizable; as noted, instances of close involvement of project officers are not commonplace but can be found in some types of funded work, and Coordinating Centers are often established for multisite studies, although with a somewhat different purpose than we have described. Clearly resources are required for these coordinating functions; however, in our experience, the efficiencies and synergies created and supported by the Coordinating Center justified the relatively modest cost of supporting one (10 percent the cost of the entire THRIVE evaluation).

More research is needed to confirm the broader applicability of, and other strategies for, addressing the challenges of mixed methods research under other circumstances of individuals, funding, and topic. Based on the THRIVE experience, the following may contribute to success:

• Consideration of not only expertise but also interest and dedication, personality, and willingness to both lead and follow others when crafting mixed-methods research team;

• Maintaining frequent, regular, and multi-modal (email, phone, in-person) communications throughout the collaboration;

• Identifying, recognizing, discussing, and addressing disciplinary and methodological differences early in the partnership (e.g., differences in the evidence needed to establish causal inference);

• Creating working guidelines and policies collaboratively, especially as related to issues of dissemination;

• Explicitly creating and maintaining nonhierarchical and collaborative leadership responsible for not only organizational and logistical needs but also for identifying and promoting discussion of current substantive issues (e.g., threats to a project's validity) and future ones (e.g., new policies or funding opportunities);

• Recognizing, divulging, and honestly choosing whether to participate in a project that will require more commitment, flexibility, and time than a less collaborative model.

Each collaboration is idiosyncratic and, as such, will experience unique challenges, requiring tailored solutions. However, the strategies identified here offer general guidance for finding solutions that permit teams to successfully mix research methods to advance health services research.

Supporting Information

Additional supporting information may be found in the online version of this article:

Appendix SA1: Author Matrix.