Process Evaluation of a Retreat for Scholars in the First Cohort: The NIH Mixed Methods Research Training Program for the Health Sciences

Abstract

The Mixed Methods Research Training Program for the Health Sciences aims to enlarge the national pool of trained investigators in mixed methods and improve the quality of grant applications to the NIH. Selected scholars are assigned a consulting team, participate in webinars, and attend an annual “retreat” focused on learning mixed methods through application to their research. Our paper summarizes the process evaluation of the retreat. Scholars identified strengths in small interactive groups to discuss individual projects and the opportunity to apply learning. Scholars wanted further opportunity to discuss individual projects, understanding interventions and mixed methods, and finding collaborators. Our findings will be useful to leaders developing workshops or similar programs at the faculty level.

Growing awareness of the need to employ mixed methods to address population and behavioral health has resulted in an exponential increase in mixed methods studies funded by the U.S. National Institutes of Health (NIH), the U.K. Medical Research Council, and in mixed methods dissertations, a bellwether for future mixed methods studies (McKim, 2015; O’Cathain, Murphy, & Nicholl, 2007; Plano Clark, 2010). A similar pattern is evident in the demand for mixed methods courses, which increased at U.S. colleges and universities (Plano Clark, 2010). Nevertheless, the opportunities to learn mixed methods at the faculty level are limited relative to learning opportunities in qualitative research (e.g., qualitative inquiry, grounded theory, ethnography) and quantitative research (e.g., statistical methods, psychometrics) (Poth, 2014). As a result, investigators often learn mixed methods through alternative pathways after completing graduate or professional education. For example, investigators may attend mixed methods workshops that often accompany major conferences, such as the British Psychological Society, the American Educational Research Association, the American Evaluation Association, the American Sociological Association, and the North American Primary Care Research Group. Universities, such as Durham and Johns Hopkins, also sponsor mixed methods short courses. While workshops and short courses provide a useful introduction to mixed methods, an intensive training program provides specific, targeted experiences for health researchers who have deficits in their skill training and who require the close attention of a skilled mentor in mixed methods and a consultant in mixed methods in their content area to improve their skills.

Numerous reports and book chapters address the topic of teaching and learning mixed methods (Christ, 2009; Creswell, Tashakkori, Jensen, & L, 2003; Earley, 2007; Frels, Onwuegbuzie, Leech, & Collins, 2012; Frels, Onwuegbuzie, Leech, & Collins, 2014), but with the exception of an article on training for mixed methods research in the health sciences in Sweden (Hansson, 2010) reports focus largely on teaching in graduate or postgraduate courses. The literature addresses overall course development, the sequence for learning mixed methods, teaching approaches, and course design. Most studies are conceptual or empirical reports (e.g., a case study) concerning a mixed methods course or series of courses at a single institution. As Frels et al. (2014) noted, the transferability of the findings of this literature to the broad number of institutions and conceptual stances is limited, and more empirical work is needed to guide instructors (Creswell, Tashakkori, Jensen, & Shapley, 2003; Poth, 2014). Aside from the structure and content of workshops or courses, Frels and colleagues highlighted the importance of a sustained mentoring relationship for career development (Frels, Newman, & Newman, 2015). Furthermore, while publications have provided guidance for writing mixed methods grant proposals (Dahlberg, Wittink, & Gallo, 2010; Saint Arnault & Fetters, 2011; Wisdom & Fetters, 2015), little guidance is available on providing training or mentoring to enhance academic survival skills of writing proposals for funding or developing mixed methods interdisciplinary teams. These skills have special considerations for investigators working in the health sciences. In this paper, we describe the perspectives of the first cohort of scholars in a national training program in mixed methods for the health sciences. This report fills a gap in the literature in providing an empirical report of a mixed methods training program intended to improve the skills, such as writing mixed methods grant proposals, of faculty in the health sciences.

The Need for Multifaceted Advanced Training in Mixed Methods

Our study addresses several practical issues linked to mixed methods training for investigators in the health sciences. First, training in mixed methods research, and the related literature to guide the process, typically focuses on courses offered to graduate students. However, faculty members often are “first generation” mixed methods researchers without the benefit of previous training of their own (Earley, 2007). Faculty members learn mixed methods through workshops, reading, and hands-on experience. Also, no existing training programs enhance workshops with individualized mentoring.

Second, the program addresses the need for mentored research training, which is not commonly addressed in the mixed methods literature (Frels et al., 2015), and not at all for mixed methods in the health sciences. Mentoring has been identified by the Institute of Medicine as key to reducing individual-level barriers to bridging the transition to independence at all career levels (Sung et al., 2003). Further evidence suggests that multiple mentoring relationships including those from outside institutions become increasingly valuable in terms of a mentee’s productivity and subjective success as academicians advance beyond their earliest stages of training and career development (Epstein & Hundert, 2002; Jones & Tucker-Allen, 2000; Ramanan, Taylor, Davis, & Phillips, 2006; Reynolds, 2007, 2008; Weinert, Billings, Ryan, & Ingbar, 2006; Zerhouni, 2005). Investigators seeking to mix methods frequently do not have role models on site with experience in designing and carrying out mixed methods projects.

A third issue is the pressing need to retain investigators in the health sciences that are often lost in their respective fields if they are unable to generate funding, potentially more difficult given the complexity of mixed methods research. The program must include early-career faculty in response to the concerns about high rates of attrition from the NIH career path by new and early stage investigators, and significant losses to the scientific enterprise (National Academy of Science, 2000; Varki & Rosenberg, 2002). The loss of new investigators from academic science has been labeled one of the most critical problems facing NIH (Board on Life Sciences Bridges to Independence, 2005), leading to a number of NIH-wide initiatives. For example, the NIMH Council (NMHAC) Report “Investing in the Future” recommended support for initiatives to “increase the return on investment” of training programs throughout the research career development pipeline and achieve an outstanding workforce by the year 2020 that can integrate novel technologies and approaches across multiple levels of analysis (National Advisory Mental Health Council Workgroup on Research Training, 2008). The need is particularly salient for training in the conduct of mixed methods research because of the complexity of mixed methods and the skillset required.

Fourth, despite these challenges, there is a need for high quality mixed methods proposals to the NIH and other funders to conduct health-related research. Although NIH-funded mixed methods studies have increased, mixed methods studies remain a small proportion of proposals funded, ranging from 0.1% to 4.9% by NIH Institute in 2007–2008 (Plano Clark, 2010), which is the most recent period available. Yet, the need remains to address behavioral health in a diverse society through approaches such as mixed methods. Enhancing training is one way to increase the number and quality of funded mixed methods studies.

Mixed Methods Research Training Program for the Health Sciences

Addressing these needs, the Mixed Methods Research Training Program for the Health Sciences (MMRTP) is a yearlong training program for researchers in the health sciences funded by the Office of Behavioral and Social Science Research with several Institutes of the National Institutes of Health participating. The application for the MMRTP was in response to a request for applications (RFA-OD-13-009, “Short Courses on Innovative Methodologies in the Behavioral and Social Sciences (R25)”). The RFA specifically mentioned mixed methods as a potential training topic. The foundation for the application of mixed methods research to NIH sponsored research traces to the 2001 NIH report on qualitative health research (National Institutes of Health, 2001) that incorporated the idea of combining methods. That report was eventually followed by the publication of the “Best practices for mixed methods research in the health sciences” in 2011 (Creswell, Klassen, Plano Clark, & Smith, 2011). The overarching goal of the MMRTP is to provide a state-of-the-art methods training program to enhance the mixed methods skills of NIH investigators.

The main outcome of the program is that scholars will prepare a grant to submit for external funding to the NIH or an organization of similar scope. Additional educational goals for scholars are to increase: appropriate use of mixed methods in research in the health sciences; attendance and presentation of mixed methods health research at conferences; authorship of publications employing mixed methods; enhanced translational research collaborations; and leadership in mixed methods at participating institutions. The significance of the program lies in enlarging the national pool of trained investigators in mixed methods, improving mixed methods skills of investigators, and improving the quality of NIH applications. The program will nationally recruit 14 investigators (called scholars) in each of four cohort-years representing diverse disciplines in the health sciences, open to faculty with doctoral degrees.

The program components are modeled after the successful National Institute of Mental Health (NIMH) Advanced Research Institute (Bruce et al., 2011) and include assignment of scholars to mentors and consultants, webinars, a website for distributing materials and discussion, and an annual retreat (the components of the program are outlined in Figure 1). Each scholar applies with a proposal that becomes the springboard for learning mixed methods during the yearlong program. The projects provide a concrete task to focus training in mixed methods, consistent with the principles of active, problem-based learning (Bruning, Schraw, & Norby, 2011). Consistent with the principles of problem-based learning and crafting learning experiences (Fink, 2013), mixed methods are best learned through application to a specific area or problem rather than solely through abstract presentations of research design or methods. Each Scholar is assigned to a mentor (one of the three co-investigators on the training grant) and to a consultant with mixed methods expertise in a related discipline (Table 1). A central feature of the program is an annual “retreat” held in the summer. The retreat is the focus of this paper.

Figure 1.

Mixed Methods Research Training Program annual cycle, carried out for each cohort, shown for one cohort. The program will serve four cohorts.

Table 1.

Key Terms

Term	Description
Program	The Mixed Methods Research Training Program for the Health Sciences that aims to provide a state-of-the-art methodology training program to enhance the mixed methods skills of NIH investigators
Scholar	Investigators training in mixed methods
Mentor	One of the Project Directors who act as primary mentor to the Scholar
Consultant	A mixed methods investigator with content expertise of the Scholar who is matched to each Scholar

Structure of the retreat

A webinar prior to the retreat provided an introduction to mixed methods research that generally followed the NIH Best Practices of Mixed Methods Research in the Health Sciences (Creswell et al., 2011). The topics covered were an introduction to mixed methods, mixed methods designs, rigorous mixed methods components, becoming a resource, and an overview to prepare for the subsequent retreat.

The retreat itself was a two and a half day event held in person at a U.S. university. It included the 14 scholars, the 3 program faculty who serve as investigators and mentors for the program, and 2 consultants. In addition, other faculty from participating institutions and the evaluator were present and gave presentations. The method of instruction was primarily lecture. We originally planned for scholar presentations of their projects each day of the retreat but ultimately removed one of those to allow extended discussion from NIH and PCORI staff. On day one and day three, scholars split into three small workgroups to discuss their specific projects. A mentor and a consultant or other faculty member led each workgroup that involved a brief presentation by the scholar followed by feedback on their grant proposal. The educational objective was that scholars would be able to explain fundamental concepts of mixed methods research and identify major elements of current mixed methods thinking to include in their project application. These elements included the justification, designs, diagrams, study aims, use of theory, sampling, integration strategies, rigorous qualitative and quantitative methods, interdisciplinary teams, evaluation topics, and strategies for writing mixed methods for proposals and manuscripts. Additional topics included presentations by NIH and PCORI panels and how to become an institutional resource. Each of these topics was tied to mixed methods research (e.g., developing aims for a mixed methods study). At the conclusion of the retreat, scholars submitted a mentoring plan that detailed the timeline to develop their grant application. The full agenda is available on the MMRTP website (http://www.jhsph.edu/academics/training-programs/mixed-methods-training-program-for-the-health-sciences/about-the-program/Retreat/).

Numerous mixed methods workshops are held regularly, yet little guidance is available on how to construct the program and what topics to cover. The retreat brings the scholars together with selected consultants to review specific topics and scholar projects. The goal of this process evaluation was to learn from the Scholars about the conduct of the retreat. Process evaluation was a critical step to understand the implementation of the program and to inform needed improvements before proceeding to outcome evaluation (Rossi, Lipsey, & Freeman, 2004). We used methods necessary to address our process evaluation questions and followed the philosophical stance of pragmatism, “the essential criteria for making design decisions are practical, contextually responsive, and consequential” (Datta, 1997, p. 34). Two research questions guided this process evaluation: What were the experiences of scholars with the MMRTP retreat? and To what extent did the MMRTP retreat carryout its goals as intended? We summarize the process evaluation of the retreat in this publication to be useful to leaders developing and teaching workshops or similar programs at the post-graduate, faculty level.

Methods

We employed a qualitative approach to the process evaluation because understanding scholar experiences and program implementation were better suited to exploration through qualitative inquiry. Open-ended data collection through observations, brief interviews, and open-ended survey items facilitated understanding scholars’ experiences and program implementation specifically for future modifications.

Participants

Fourteen scholars participated in the mixed methods research training program. All of the scholars were researchers, assistant professors, and associate professors at institutions in the U.S. The institutions represented were geographically located on the West Coast (3), Midwest (1), South (3), and East Coast (7). Disciplinary orientations were: general medicine, nutrition, human development, music therapy, psychology/psychiatry, oncology, pediatrics, and social work. Thirteen of the 14 scholars were women. Scholars were actively preparing grant applications targeted for the following funding mechanisms: NIH R01 awards, NIH K series career development awards, and Patient-Centered Outcomes Research Institute (PCORI) awards. Three reported they were primarily trained qualitatively, and the remaining were primarily trained quantitatively. Two indicated some prior mixed methods training.

Data Collection for the Process Evaluation

We collected data through observations during the entire program retreat, mini-interviews with scholars, and the Mixed Methods Scholar Self-Assessment (the subject of other manuscripts (Guetterman et al.) The program evaluator (TG), who is not a mentor in the program, collected observations throughout the retreat using an observational protocol. The reason for observations was to allow real time documentation of topics discussed, questions from scholars, and interaction and to eliminate the need to rely on recall of events after the retreat. Each session yielded observations. Because the workgroups were split into three, the evaluator rotated and spent 40 minutes observing each workgroup. At the end of the retreat, we also held a brief group discussion to gather scholars’ reflections. It was not practical to gather audio or video recordings through the retreat given the number of individuals present and multiple rooms in use. Nevertheless, the observational data yielded extensive written fieldnotes based on interactions with the scholars and data collected throughout the retreat. The evaluator then wrote observational memos each day of the retreat to summarize observation. In addition, the evaluator conducted brief semi-structured mini-interviews with all scholars throughout the retreat to understand their experiences and suggestions for improvement. The mini-interviews lasted 10–15 minutes. They were not audio-recorded because they occurred at unscheduled times and locations not conducive to recording, but the evaluator took detailed notes. The semi-structured format allowed the evaluator to probe topics and provided valuable spontaneous insights from individuals. We also gathered process feedback through open-ended survey questions about goals and reflections on the retreat itself. These questions allowed us to probe scholars’ experiences after they had time to reflect on the retreat and were efficient to administer as part of the self-assessment. Table 2 provides a summary of questions included in the data collection protocols.

Table 2.

A Summary of Questions in Data Collection Protocols

Questions Guiding Observational Protocol

What is the interaction among scholars, mentors, and consultants? In what ways are scholars making their project more likely to be successful or fundable? What topics were covered (inside and outside of sessions) that were not on the agenda? What did participants seem to find helpful or not? How are participants helping other scholars based on their expertise? How did the scholars incorporate ideas from prior sessions at the retreat? How did the time allocation work out?

Mini-interviews

How has the retreat been going for you? What could be done differently?

Open-ended questions on skills assessment form

Describe your goals for the Mixed Methods Research Training Program. What skills and goals are most important to you? What would you like to learn? What aspects of the retreat were helpful? What would you have liked to change about the retreat?

Data Analysis

We conducted qualitative theme-based text analysis (Kuckartz, 2014). Each scholar was the unit of analysis though data came from observations, mini-interviews, and the skills assessment form as previously noted. The evaluator (First Author) led the analysis and coded the data. All four authors participated in the analysis by reviewing findings. Following Kuckarz (2014), step 1 was to read the text database and make notes of important passages. We recorded notes and brief memos about interesting aspects. Next, the evaluator developed general categories of interest based on the goal of conducting the analysis of program data. The goal was to categorize what happened during the program (i.e., the process) and what needed to change (i.e., outcomes). The next step was to code all of the data based on codes that emerged through the text (i.e., none were set in advance) and began grouping some codes into themes. Through this process, the evaluator created a codebook that defined each code. We then compiled the coded segments for each code and began grouping codes that represented related ideas into themes. The next step was to review all of the data using the complete coding system and revise and rearrange codes to portray a narrative of the retreat process and evaluation. All authors, including mentors, then participated in the review of findings. We used MAXQDA, a qualitative data analysis software application, to manage the data and codes.

We implemented three validity checks to check the accuracy of findings. First, we used triangulation (Denzin, 1978) through collecting multiple data sources, including observations collected through the entire retreat, and incorporating all into the analysis. Next, we looked for disconfirming evidence of themes through the dataset to ensure an accurate representation of each (Creswell, 2013, 2015). Finally, because the evaluator led the analysis and the program mentors also reviewed findings, we implemented peer debriefing by asking two individuals external to the project to review the findings (Lincoln & Guba, 1985).

Results

Overarching Interest in Mixed Methods

The strong interest in mixed methods and the program was an overarching theme, as all scholars expressed their desire to both learn mixed methods and learn to improve their grant writing. In addition to individual interest, support from the individuals’ home departments was evident. In a mini-interview, a scholar described support from department and chair, specifically, who offered additional resources, such as purchasing textbooks. During the retreat two scholars discussed department interest, and one said “my department is excited and asking about sharing resources, having a brownbag or a seminar.” The retreat cultivated interest in becoming a mixed methods resource within scholars’ home institutions, based on observations of several scholars who expressed confidence that they could provide guidance about designs and enhance the proposals of colleagues by applying what was learned in the retreat.

Scholars expressed that many aspects of the retreat process worked well to enhance the skills of scholars and develop their projects. Table 3 provides a summary of themes from the retreat and sample excerpts of data. We organized the exposition of the themes under three rubrics: academic survival skills, specific mixed methods topics, and the interactional nature of the retreat. In the following summary, the source of data was fieldnotes and observations unless otherwise specified.

Table 3.

Themes Derived from Process Evaluation of the Retreat

Theme	Description	Illustrative quote or observation
Academic Survival Skills
Writing grants	Scholars noted they appreciated seeing grants and working on grants. They wanted more of this activity.	“Presentation of examples of actual proposals and papers”
Writing for publication in medical journals	The faculty discussed structuring the mixed methods papers.	A scholar’s goal “To eventually publish journal articles reporting on mixed methods study”
Finding collaborators and joining academic teams	Scholars reported value in connecting with others to collaborate on research.	“Relationship with mentors, consultants, and other scholars in order to collaborate in the future - and also get feedback on specific proposals”
Specific Mixed Methods Topics
Focus on mixed methods	Scholars suggested tying more aspects to specific mixed methods issues.	“A more sustained focused on mixed methods research throughout the retreat”
Intervention designs	Scholars expressed a strong interest in advanced applications of mixed methods to intervention studies.	“How to conduct a rigorous mixed methods intervention study starting from development till analysis and outcome assessment”
Integration methods	Integration of qualitative and quantitative results was a persistent interest of all scholars.	“How to actually combine qualitative with quantitative approaches”
Conceptual frameworks for projects	Applying learning refers to instances in which the scholars applied something in a previous session to a current session or their project	A scholar brought up diagrams and tables to relate the design to reviewers.
The Interactional Nature of the Retreat
Interaction among scholars, mentors, consultants	Individuals interacted during the retreat, providing feedback.	The mentor and consultant both asked questions about the study
“Small groups were key”	Several iterated the importance of the small workgroups and wanted more workgroups.	“Small group interactions to discuss individual projects were very helpful to modify and refine my proposal outlines”
Focus on individual projects	Scholars wanted more time to talk about their own projects.	“It would be helpful if more time was dedicated to workshopping projects, and that it happened daily; perhaps in rotation with the different mentors”
Improving projects and helping each other	Scholars provided feedback to each other drawing from their expertise.	“It was also very helpful to benefit from peer mentoring during the breakout groups, which generated rich (if brief) discussions.”
Involvement of federal funding staff	Nearly all scholars commented on the value of having PCORI and NIH officers at the retreat.	“The presentations from funding agencies (especially PCORI) were great, although in the future they could limit their remarks about large multimillion dollar FOAs that are less relevant for this junior faculty audience.”

Note: Data from the Mixed Methods Research Training Program for the Health Sciences, 2015.

Academic Survival Skills

Value of a focus on writing grants

A key finding related to writing a grant proposal that uses mixed methods, a primary focus of the retreat. Nearly all scholars commented in the survey on the value of seeing actual grants, such as “Seeing examples of grants were also very helpful.” In the main sessions, faculty shared examples of both K and R series grants by discussing the grants projected on a screen. Scholars learned about ways to structure the grant, the layout, the inclusion of visuals, and specific writing techniques that are helpful from the perspective of the reviewer. For example, a consultant presented a strategy to include everything possible on the environment to ensure a good score for that section. A scholar reported in the survey that she “REALLY [emphasis in data] appreciated the review of NIH grant applications,” and most provided similar comments. Furthermore, the workgroups allowed more time to address the grant strategy. This topic included the sequencing of grants into a program of research and using the appropriate mechanisms. For example, a consultant offered that following a K award, an R34 mechanism can be used for developing an intervention followed by an R01. Feedback also related to the significance of the study, as a consultant posed, “What’s the new splash?” The consultant then made a point about the importance of significance scores in the review. In many of the sessions, consultants and mentors provided feedback to develop a “strong effect” intervention. The focus on grant writing seemed beneficial based on scholars’ incorporation of ideas and their description of the focus on writing grant proposals as “very helpful” in their survey responses. However, nearly all scholars commented that they wanted more time in the retreat to work on grant writing.

Writing for publication in medical journals

In the survey responses nearly all scholars cited the need to gain skills in writing mixed methods manuscripts for “publication in high impact journals.” Scholars also wrote about goals to “present mixed methods studies at national and local conferences.” The retreat touched briefly on the topic of writing, including planning publications at the start of the study. In the survey, nearly all scholars described writing a mixed methods paper as something they would like to learn or suggested enhancing the topic at the retreat.

Finding collaborators and joining academic teams

Finally, scholars described needs related to working with teams on a mixed methods study and “finding collaborators who fill gaps in methodology” in their survey responses. They also wrote about importance of face-to-face meeting with mentors the retreat (e.g., “great opportunity to get to know the mentors”). Some individuals suggested in the survey that more time for networking would be helpful at the retreat itself. The observations corroborated this point. Specifically, many sessions ran over on time, encroaching on the little time that was available for scholars to talk and network among each other. The addition of time to network could further develop this community of learners into a community of mixed methods scholars.

Specific Mixed Methods Topics

Focus on mixed methods

Another theme from the retreat related to specific research methods skills. A set of comments in the survey requested the retreat sessions be more mixed methods focused. A scholar noted, “presentations that are general to research should not be included.” Additional suggestions from surveys were to connect certain discussions (e.g., interdisciplinary teams, evaluation) to specific mixed methods issues. Furthermore, nearly all scholars requested being able to attend “both of the qualitative and quantitative sessions.” One scholar suggested providing these sessions before the retreat if both cannot be included.

A mixed methods skill that warrants particular focus is integration of qualitative and quantitative research. Based on observations, mini-interviews, and survey responses, scholars expressed a strong need for learning about approaches to integration. First, the topic of visual displays seemed new and of interest to many. Visual displays include the use procedural diagrams to represent the research design and the use of joint displays for integration. Scholars cited it in the survey as an important skill to learn along with “rigorous methods for integration.” Several suggested more time for advanced topics of integration and interpretation, which they felt was helpful. In survey responses, scholars noted they would have liked more time for the topic of integration and requested it for extended discussion in follow-up webinars.

Intervention designs

Other skills covered were broader than mixed methods research and concerned interventions and qualitative inquiry. First, developing intervention designs seemed to be a popular topic during the retreat. Many scholars had questions about these designs and suggested sessions in future retreats. Second, numerous questions arose about qualitative special topics, such as data collection, coding methods, analysis, and intercoder agreement. Frequent questions from scholars at the retreat suggested the need to gain experience with qualitative designs, such as how to incorporate ethnography or grounded theory into an intervention. Another scholar inquired about conducting interviews during an intervention without influencing the outcome. Moreover, in the survey many scholars noted the need to improve understanding of sampling in mixed methods and purpose sampling in qualitative research.

Conceptual frameworks for projects

Throughout the retreat, we observed evidence of the scholars applying what they were learning through using various conceptual frameworks in their projects. In many instances, scholars brought something from a previous session to a current session or their project. For example, during a workgroup presentation, scholars suggested to a peer including a procedural diagram and a table in the grant to more easily relate information. That topic was discussed in an earlier session in which scholars looked at actual grants. In the second workgroup session, several scholars began their presentation with changes that reflected what they had learned. Examples were rewriting study aims and revising procedural diagrams with additional detail. Another scholar developed a table linking aims to methods to clarify her design. Sometimes, mentors facilitated learning by helping scholars make connections to previous topics and discussion.

The Interactional Nature of the Retreat

Interaction among scholars, mentors, and consultants

Another theme of the retreat was the interaction among scholars, mentors, and consultants. Overall, scholars seemed very eager and receptive to feedback, likely because most of the interaction among scholars, mentors, and consultants centered on improving their projects. Although interaction was somewhat low initially, it increased through the retreat. Initially, only mentors and consultants offered feedback, but by the final day scholars also provided feedback to each other. Some of the feedback was related to general career development. A scholar noted “hearing their experience and their own career trajectories from mentors and consultants attending” was valuable. Other feedback was more specific. In the workgroups, the mentors and consultant suggested ways to make the project more fundable. Suggestions included working with a local clinical and translational science center and determining the correct size and scope of the study. In another example, the consultant provided feedback on the grant content and the issue of surface culture versus deep culture in the investigation. The consultant explained that surface culture is concerned with customs and outward behavior, but deep culture is concerned with the thoughts, beliefs, values, and interpersonal interactions associated with culture. They discussed how qualitative inquiry allows exploration of deep culture.

In another example of scholars improving their project during the retreat, the workgroup group discussed the value of the Consolidated Framework for Implementation Research (CFIR) for a scholar’s project. The CFIR is an overarching typology to guide the development of theory for interventions and examination of what works based on five domains: intervention characteristics, outer setting, inner setting, individuals involved, and the implementation process (Damschroder et al., 2009). The discussion in the workgroup centered on “whether CFIR provided sufficient detail” as a conceptual framework for the project. Because the scholar’s project involved the development of an intervention in primary care, the discussion helped the scholar to understand how theory can be integrated into her mixed methods study. In addition to mentor input, scholars used their expertise to help each other. One scholar commented in the survey, “It was also very helpful to benefit from peer mentoring during the breakout groups, which generated rich (if brief) discussions.”

Small groups were key

In reflecting on the retreat, the majority of scholars reported, “small groups were key,” or very similar language referring to the two workgroup sessions of four to five scholars, led by a mentor. In these workgroups, each scholar had time to present their project and solicit feedback. A scholar wrote in the survey, “small group interactions to discuss individual projects were very helpful to modify and refine my proposal outlines.” The scholars observed in the workgroup seemed very receptive to the ideas. They asked questions and seemed excited about the ideas, as ways to improve the grant. Four scholars reiterated the importance of the small workgroups and wanted more workgroups. One scholar suggested rotating mentors in the small groups to provide different perspectives.

Focus on individual projects

A potential area to change related to focusing on projects. Participants generally wanted more time to talk about their own projects and receive feedback. Many commented in the survey that dedicating more time for projects was important even if it required lengthening the retreat. Other program suggestions from mini-interviews included ways to use webinars or future retreats for mock NIH study sections by type of award (e.g., K focus). In general, scholars requested more time, such as adding one to two days, to “allow for more small group discussion…and writing.” Observational data showed that many sessions ran out of time as the schedule was pushed back. In the scholar survey comments, several complained about “anecdotes” interfering with the schedule. Related to timing is the overall sequence of learning. One scholar noted that portions of the retreat repeated, “what we’d learned in our webinar.”

Involvement of federal funding staff

Nearly all scholars commented in surveys on the value of having NIH and PCORI at the retreat. For example, scholars noted “hearing from POs” (program officers) was “especially beneficial.” The NIH representatives gave a presentation and fielded many questions about funding mechanisms and grant applications. The session with PCORI representatives also seemed useful to scholars. For instance, the PCORI representatives discussed ways to enhance an evaluation project to make it more fundable. Between sessions, many connected with both the NIH and PCORI representatives to discuss their individual projects.

Discussion

The MMRTP is unique because no other national program in mixed methods education and mentorship exists for the health sciences. Involving all three mentors and several consultants in the retreat provided a starting point for the mentoring relationship and project development. The scholars then work on their grant applications with mentors and consultants over the course of the year in the program, and the retreat served as a venue for mentored work on the scholars’ projects and to create a network and “community of scholars” in the broad sense. The findings of the evaluation of the retreat process provided insight into the program and ways to improve. Insights relate to the content of the retreat and the overall focus. Individuals developing mixed methods training programs can use the findings of this report, but it should be particularly useful for programs focused on developing proposals for funding in the U.S. and internationally. For discussion purposes, we refer to the retreat as it was our term for an in-person training meeting. However, it the findings provide insight for planning a mixed methods workshop, short course, or other training program.

The Content of the Retreat

Beginning with the content, scholars mentioned concerns about the topics covered and their sequence. To address that concern, using webinars and pre-retreat reading assignments could ensure a baseline level of understanding, and the retreat could be used “to go deeper into material.” These preliminary learning activities develop the critical foundational mixed methods knowledge so that learners can focus on their own projects and advanced topics at the in person training. The scholars had clear opinions about the practical aspects of positioning manuscripts and research proposals employing mixed methods to be successful in gaining publication and funding. While circulating exemplary publications addressing contemporary issues in mixed methods was useful (e.g., as found in Curry and Nunez-Smith (2014)), example mixed methods proposals for funding with the opportunity to speak to the principal investigators would be very helpful. By having scholars work on their funding applications through the retreat with multiple opportunities to obtain feedback, scholars would exit the retreat with a reasonably developed grant. In order for that to be realistic, scholars would have to come into the retreat with a developed outline of their project and then learn about essential mixed methods components that would enhance the rigor and sophistication of their proposal. This process can begin with walking scholars through an existing proposal that included mixed methods in order to literally see where these components fit. Finally, having scholars review the NIH Best Practices Checklist (Creswell et al., 2011) would provide insight into what NIH study section reviewers might be looking for in their proposal. While sponsored by the NIH, the checklist can apply broadly to any funding proposals in the health sciences.

A necessary caveat is that mixed methods remains an emerging field. For instance, scholars asked questions about advanced topics, related to intervention designs, and qualitative special topics arose, such as incorporating ethnography or grounded theory or conducting interviews during an intervention without influencing the outcome. How to incorporate mixed methods into intervention development or in implementation research, however, reflect issues the field needs to address as a whole. The topics are important research questions as well as educational topics that will help scholars develop the foundations to conduct mixed methods research for the health sciences.

The Overall Focus of the Retreat

The second major insight involved the overall focus of the retreat. Overall, it seems that moving to a more applied, project-based workgroup would be helpful to scholars. Although the balance between methods and content can be difficult, Greene (2010) reminded us, “Methods are ever the servant of inquiry purpose, never the master” (p. 4). In that vein, the program was designed such that the scholars’ projects would be a platform to learn mixed methods. Their feedback indicated that this approach is the right idea but deserves more emphasis. During project discussions, mentors and consultants had ample opportunity to provide education about the practical aspects of mixed methods. For example, during a scholar’s project presentation, mentors could identify mixed methods components that need enhancement in the proposal and provide a specific suggestion for what to add to strengthen the proposal.

Sustainability of the scholars mixed methods skill development after the retreat will be critical. Follow-up webinars that combine presentations on mixed methods topics with hands-on work and examination of scholars’ projects are one way to foster development. Shifting follow-up learning online uses fewer resources and provides a booster to keep the learner engaged in mixed methods. Additionally, ongoing mentoring with active participation from both the scholars and mentors, is needed to foster collaboration with research teams and transformation into an autonomous research, as suggested by Frels et al. (2015). Support from the home departments of scholars was evident. Institutional support will facilitate scholars becoming a resource in their individual universities and organizations, building a national cadre of investigators with training in mixed methods.

Lessons for Moving Forward

The process evaluation of the retreat yielded important lessons concerning grouping the scholars, content to include, and scholar presentations that we plan to integrate into training of future cohorts. Regarding learner grouping, the selection of scholars should be heterogeneous in terms of funding goals to allow scholars to be grouped in the retreat based on their funding goals. Selection would require scholars that represent a combination of those desiring career development awards and those with career development awards who want to develop research grants. Other lessons learned relate to the content of the webinars and retreat. First, we plan to share more grants and feedback received from reviewers. Along those lines, consultants’ presentations should focus on examples of their grants and manuscripts, as scholars responded positively to examples shared and wanted more included. The specific grants shared should match the audience (e.g., European Commission Research & Innovation examples may fit European audiences and NIH grants for US health sciences audiences). Next, it seems important to move more content (i.e., didactic portions) to the webinars. The webinars can be further used to begin discussing the scholars’ projects. Moving content would free time for the scholars to present each day of the retreat. That change would further promote active learning and keep the focus on scholars’ projects. The small group format worked well but may be further enhanced by rotating mentors through each group so that scholars reap the benefit of feedback from multiple mentors.

Based on the findings, we recommend an approach in which scholar presentations build over the three days of in person learning. On the first day, scholars will introduce their topic as everyone is still figuring out what everyone is doing. On the second day, presentations will continue, but we would expect more scholars giving feedback to each other about their projects. Finally, on the third day, scholars will prepare for next steps (we missed this in our first retreat by only having two sessions), such as goals for writing their grants and connecting with consultants.

Limitations, Future Research, and Unique Contributions

Our study is based on the 14 scholars in the first cohort of the MMRTP and the retreat. However, the scholars represent major research institutions throughout the U.S. The findings are likely transferrable to other mixed methods training programs targeting faculty-level scholars. Another potential limitation is that we did not record interviews. We instead relied on detailed fieldnotes for analysis of the three data sources, impromptu comments from scholars, and prolonged engagement throughout the retreat (Onwuegbuzie & Leech, 2007). Researcher bias is a potential limitation that we attempted to mitigate by having an evaluator, who is not a program mentor or investigator, conduct the primary analysis (Onwuegbuzie & Leech, 2007). Although this paper presented a process evaluation of the retreat, substantial time needs to elapse to measure outcomes of the program such as grant success, presentations, and published papers among the scholars. Future research should examine long-term performance of the scholars, similar to the evaluation of the NIMH Advanced Research Institute (Bruce et al., 2011), which examined the successful funding of participants.

The unique contribution of this study lies in its focus on a mentoring-based mixed methods research training program to train faculty-level scholars. It provided insights into the process of such a training program and what we learned. While the current literature generally covers teaching mixed methods courses (Frels et al., 2014), this study is unique because it provides empirical evidence of an in-person training situated within a larger program and focused on writing grants. The findings may be applied to other mixed methods programs throughout the world. Specifically, the findings emphasized the value of a hands-on, interactive approach to developing mixed methods skills. Designing the training program using a grant project as a platform for learning mixed methods is a way to enhance skills and ensure that scholars exit with a better developed proposal (i.e., a tangible product) to refine with ongoing mentored support.