Don't be afraid to eat the whole whale! Using mixed methods to enhance what we learn in leadership research and assessment

Abstract

The purpose of this article is to identify proper uses for mixed methods in leadership research and assessment. This article will highlight research and assessment questions that are best served by mixed methods and will offer practitioner‐friendly guides and examples for integrating quantitative and qualitative data to help eat the whole whale “one bite at a time.” Sound decision‐making in the design, execution, and presentation of mixed methods studies and outcomes assessments not only produces better outputs, but greatly increases the likelihood of (a) publication in reputable journals, (b) sound program evaluation decisions, and (c) public consumption of study or assessment results.

Because leadership is a complex social science phenomenon, the “more is better” data collection philosophy can often be adopted when designing studies or program assessments. But when is “more” really more? Merely having multiple forms of data does not automatically equate to a better study or program assessment. To further advance the discipline of leadership and leadership development using sound methodologies (Riggio, 2013), it is incumbent upon leadership researchers and program administrators to understand and articulate (a) when a research problem or program assessment is best served by mixed methods and (b) how to maximize contribution through rigorous execution of quantitative and qualitative integration practices.

Mixed methods are a good fit when one data source will be insufficient for addressing a research problem or assessing a leadership program. Specifically, mixed methods are desirable and justified when (a) research or assessment results need to be more complete and/or corroborated, (b) initial results need to be explained, (c) exploratory research or a needs assessment will be necessary prior to administering instruments, (d) exploratory findings need to be tested in order to be generalizable, (e) different types of cases need to be compared and contrasted, and/or (g) participants need to be involved in the study or assessment design (Bryman, 2006; Creswell & Plano Clark, 2018).

DESIGN IS NOT THE BACKSEAT DRIVER

If we really want more data to be more, design must be in the driver's seat. Clear research problems and program assessments that are well justified for the use of mixed methods (Bryman, 2006) should lead to well formulated research or assessment questions that are specifically answered by multiple forms of data and a particular mixed methods design. A well justified mixed methods design then allows the researcher or program administrator to document and follow sound and tested procedures for integrating quantitative and qualitative data within analysis and presenting results.

Convergent mixed methods design

There are three core mixed methods designs: (a) Convergent, (b) explanatory sequential, and (c) exploratory sequential (Creswell & Plano Clark, 2018). In the convergent design (the design formerly known as “concurrent”), quantitative and qualitative data are collected concurrently so the results can be compared or combined (see Figure 1). The intent of a convergent design can be to compare results to obtain a more complete picture of the research problem or outcomes assessment, confirm or disconfirm one set of results with another, and/or validate measurement by ascertaining if similar responses come in both quantitative and qualitative forms (Creswell & Plano Clark, 2018).

FIGURE 1.

Convergent Mixed Methods Design (Adapted from Creswell & Plano Clark, 2018)

Convergent design research example

Taylor et al. (2011) utilized convergent mixed methods design principles to examine leadership processes within six urban water management agencies. While the focus of the study was qualitative in nature (in fact, the authors indicated in the article that they chose a “multiple case study research design,” p. 416), qualitative data and quantitative data were collected concurrently. Individual interviews were conducted with six identified water management agency “champions” as well as nominated peers in other leadership roles. Additionally, a multi‐rater questionnaire was also administered for each interviewed individual. While the interview protocol was designed to examine contributing factors to leader emergence and effectiveness of these “champions,” the multi‐rater questionnaire was designed to test the relative importance of certain literature‐derived traits and behaviors that seemed germane to the “champions” work.

Convergent design program evaluation example

At the University of Nebraska—Lincoln, there is a 70+‐year‐old leadership mentoring program comprised of 180 college student leaders who mentor 180 K—12 student leaders in the local community, called NHRI Leadership Mentoring. Program outcomes, objectives, and leadership competencies were mapped a few years ago to articulate the competencies required to meet the program outcomes, how those competencies are developed through program objectives, and how the accomplished objectives lead to the achievement of program outcomes.

To evaluate the NHRI program's effectiveness, two assessments were created with the intention of administering them after NHRI's Annual Recognition Day each April. The first assessment (adapted from Seemiller, 2014) was designed to gauge perceived growth in each of the targeted leadership competencies. To examine behavioral proficiency among the targeted leadership competencies, a second assessment was created to self‐assess perceived proficiency on the program outcome statements as well as to provide open‐ended commentary on (a) which program outcome area grew the most and why and (b) which program outcomes were reflected in their final year‐end project. The qualitative questions associated with the second assessment provided additional depth and insight to the quantitative data by elucidating student perception of growth due to their leadership mentoring experience.

Explanatory sequential mixed methods design

In the explanatory sequential design, there are two distinct and sequential, but also interacting, phases (Creswell & Plano Clark, 2018). The first phase involves the collection and analysis of quantitative data, and the second phase involves the collection and analysis of qualitative data. The intent of the explanatory sequential design is to utilize qualitative data to explain or expand upon the quantitative results (see Figure 2).

FIGURE 2.

Explanatory Sequential Mixed Methods Design (Adapted from Creswell & Plano Clark, 2018)

Explanatory sequential research example

In 2019, my graduate assistant (at the time), Hannah Sunderman, and I wanted to examine the relationship between generativity and socially responsible leadership among college student leaders who mentor (Hastings & Sunderman, 2019). Alice Rossi in 2001 conducted a seminal midlife development study where generativity (i.e., care and concern for establishing and guiding the next generation; Erikson, 1950, 1963) emerged as the strongest predictor of social responsibility. In 2015, we studied generativity in young adults and found that college student leaders who mentor demonstrated significantly higher generativity levels than their peers (Hastings et al., 2015). In the wake of the 2015 study, we wanted to see if Rossi's (2001) results translated to a young adult population, but more specifically to socially responsible leadership. Since, generativity is considered a midlife construct and since we were extending Rossi's (2001) work to study socially responsible leadership, we anticipated the necessity of multiple forms of data. Thus, the first, quantitative phase examined the predictive relationship between generativity and socially responsible leadership among college student leaders who mentor using multiple regression. Again, because generativity is considered a midlife construct, its associated measures are designed for midlife respondents. Thus, we felt it was important to gain additional insight to the quantitative findings, so the second, qualitative phase used a phenomenological design to explain and provide depth to the quantitative results by conducting semi‐structured interviews among a sub‐sample of the quantitative phase participants.

Explanatory sequential evaluative example

Currie et al. (2009) studied the influence of the institutional environment on secondary schools in England utilizing an explanatory sequential mixed methods design to compare an emergent government‐prescribed results‐oriented leadership approach with a traditional professional value‐based approach within a bounded system of secondary schools. The quantitative phase compared the two leadership approaches as well as examined the influence of school context (e.g., geographic location, size, financial situation). In the words of the authors, the follow‐up qualitative phase “enabled us to better interpret these findings and to examine the enactment of leadership”—a quintessential statement for the intended purpose of explanatory sequential mixed methods designs!

Exploratory sequential mixed methods design

In the exploratory sequential mixed methods design, there are also distinct and sequential, but interacting phases (Creswell & Plano Clark, 2018). The first phase involves the collection and analysis of qualitative data. The results of the qualitative phase are typically utilized to determine the quantitative measure, the generation of variables to test, the creation of a quantitative instrument, the development of an intervention, and/or the development of a product. The quantitative phase then tests what was developed from the qualitative phase. The intent is to interpret how the quantitative results build from the qualitative results or how the quantitative results provide a clearer understanding given their grounding in initial qualitative data (see Figure 3).

FIGURE 3.

Exploratory Sequential Mixed Methods Design (Adapted from Creswell & Plano Clark, 2018)

Exploratory sequential research example

In 2014, I gathered a group of undergraduate research assistants, graduate students, and faculty colleagues to commence a multi‐year research effort to study leadership transfer in rural communities (specifically, transfer of community leadership roles) using an exploratory sequential research design (Hastings et al., 2021). We had noticed some transfer trends that were likely to disproportionately affect rural communities. For example, the United States was estimated to experience a $75 trillion wealth transfer from older to younger generations between 2010 and 2060 (Macke et al., 2011). Simultaneously, Baby Boomer were expected to retire at a rate of 10,000 each day until 2030 (Martinek, 2008), resulting in a substantial transfer in leadership constituting 55% of managerial positions as well as 640,000 not‐for‐profit executive positions (Tierney, 2006; U.S. Bureau of Labor Statistics, 2017). We knew that the current sustainability efforts within rural communities could be markedly impacted by such transfers, bringing vitality or destruction.

At the time, there was no substantial literature and/or theory to guide the study of leadership transfer in rural communities. Thus, the purpose of our exploratory sequential mixed methods study was to identify themes related to rural leadership transfer using grounded theory and to test the facilitation of effective leadership transfer using structural equation modeling (Hastings et al., 2021).

To identify themes related to rural leadership transfer, we started qualitatively with the help of community development organizations. Five rural community development organizations unanimously nominated three communities that had a track record of successful community leadership transfer. Data were collected via semi‐structured interviews with adult and youth leaders within the three nominated communities and were analyzed using grounded theory processes (Strauss & Corbin, 1998). The emergent model of leadership transfer from the qualitative phase led to a series of propositions that were tested via structural equation modeling using data from a statewide rural poll. In quantitative research, we typically start with a hypothesis to test, and we look to previous literature or research to establish this a priori hypothesis. In the case of this study, we did not have substantial literature or theory on community leadership transfer, so we needed to utilize qualitative inquiry to develop sound hypotheses to test.

Exploratory sequential example in leader/leadership development practice

One of my favorite ways to use exploratory sequential mixed methods in leadership education is in the design of leader/leadership development (LD) programs. Effective design and delivery of LD programs require theoretical foundations. LD interventions, irrespective of type, tend to have a positive impact on a variety of outcomes (affective, behavioral, cognitive, and organizational performance); however, the impact of LD interventions differs, in part, based upon the theoretical foundations (Avolio et al., 2009). Historical approaches to LD involved finding the “silver bullet” leadership theory and model (e.g., transformational, servant) and train all leaders in that theory. This approach, over time, has not tended to document its intended results and has, thus, drawn extensive criticism in the field (Day et al., 2014; Day & Liu, 2019). Instead, recommended contemporary LD practice is to tailor LD programs to the developmental needs of participants (Day et al., 2014).

Assessing LD needs prior to intervention involves a two‐phased process. The first phase (which I call the “Diagnostic Phase” with my graduate students, but feel free to pick your favorite descriptor!) is the initial process whereby you get a lay of the land. This diagnostic phase is a prime opportunity to utilize qualitative methods such as interviews and/or observations to identify your organization/community's salient leadership issues and the needs from a leadership intervention. Qualitative analysis from this diagnostic phase data can determine which leadership theories apply.

Then the second phase is what I call the “Leadership Assessment” phase (but, again, use your own favorite descriptor!), because this is where you can utilize quantitative methods to collect baseline leader/leadership data that will inform the design and delivery of the LD intervention. For example, this might involve administering psychometric assessments associated with the chosen theories, analyzing that data, feeding back the analyzed data, and co‐creating the LD intervention with the client or organization based on the leadership assessment phase results.

AHA! IN ANALYSIS: WHY MIXED METHODOLOGISTS HAVE MORE FUN

The great beauty in using mixed methods is the rich “Aha!” moment derived from integrating multiple forms of data. The key with integrating quantitative and qualitative data in mixed methods research is to not “let the tail wag the dog,” or more accurately, to recognize which part is the tail, which part is the dog, and to have the correct part wag the other. Taking the time to design well and justify a mixed methods study or program assessment affords you the freedom and opportunity to use sound and tested procedures for integrating quantitative and qualitative data within analysis and presenting the results.

Analysis in convergent designs

Integrating quantitative and qualitative data looks differently in each of the three core mixed methods designs (Creswell & Plano Clark, 2018). For the convergent design, integrating quantitative and qualitative data can take on four forms: (a) quantitative and qualitative data collected simultaneously, but are analyzed independently then synthesized and compared at the end; (b) one form of data is transformed into another; (c) questionnaire items are asked in both open and closed formats and responses are compared for measurement accuracy; and (d) both forms of data are integrated while they are being implemented.

In the Taylor et al. (2011) study example of champion leaders in water management discussed earlier, the authors utilized multiple integration strategies in their analysis, ranging from data transformation to integrative comparison of interview and survey data. For example, after the qualitative interviews, the lead author utilized interview notes and memos to categorize and count references to different leadership styles (data transformation). Later in the analytic process, interview data and multi‐rater survey data were integrated and compared to identify key behaviors that distinguished high‐performing “champions.”

Regarding the NHRI program evaluation example highlighted above, the concurrent collection of quantitative and qualitative data afforded the opportunity to utilize the assessment data more effectively for program evaluation decisions. For example, the quantitative assessment results indicated strong perceived growth in targeted leadership competency statements as well as behavioral proficiency on the targeted leadership competencies. Additionally, each program outcome area had a calculated average value above threshold year over year for determining whether overall self‐perceived proficiency was being targeted and developed through the NHRI experience. Had we only collected the quantitative data, we perhaps would have walked away saying, “NHRI meets its intended outcomes and objectives. Woo hoo!” The qualitative data provided a much stronger and more holistic view of student experience that led to valuable program evaluation decisions. For example, one of the program outcome areas, Provide active listening, feedback, and/or guidance to sharpen the actions and thoughts of others, had the highest average level of perceived proficiency over a 2‐year evaluative period and was identified by the highest percentage of students as the most significant growth area. Specific mention of an interpersonal leadership class as well as small group meetings for mentors among the qualitative results suggested that students connected the outcome area with their training and reflection opportunities. Conversely, the program outcome statement receiving the lowest proficiency ratings, Reinvest personal leadership strengths, values, and skills for the purpose of positive social change, had, consistently, the lowest‐rated associated competency (social responsibility) with the highest variability. This program outcome also had the lowest percentage of students who identified the outcome area as the area in which they grew the most. Evaluating this assessment result led to a change in retreat curriculum to focus on the social change model of leadership development (SCM; Higher Education Research Institute, 1996) to create stronger connections between students’ mentoring experience and its societal benefit.

Analysis in explanatory sequential designs

For the explanatory sequential design, there are two points of integration for quantitative and qualitative data. First, the quantitative results often inform how the qualitative data will be collected (who will be sampled in the qualitative phase and/or what questions will be asked in the semi‐structured interviews). Second, once the qualitative data are collected, the quantitative data and qualitative data will be integrated by utilizing the qualitative results to provide insight to the quantitative findings.

In the research example above where we wanted to examine and explain the relationship between generativity and socially responsible leadership (Hastings & Sunderman, 2019), our analytic challenge was to identify how the qualitative results explained the quantitative findings. The quantitative phase results indicated that 27% of the variability in socially responsible leadership was explained by generative concern, generative behavior, and generative commitment. Generative concern, however, emerged as the only significant predictor, suggesting that higher socially responsible leadership was associated with a higher demonstration of generative concern. Although the bivariate correlations between socially responsible leadership and generative behavior and generative commitment were significant, generative behavior and generative commitment did not contribute significantly to the regression model.

Through the qualitative phase, we learned from the sub‐sample of collegiate leadership mentors that serving as a mentor provided the necessary context to be more conscious of generativity. The act of mentoring brought about generative awareness, which then revealed more of the positive effects of socially responsible leadership, creating a cyclical relationship whereby generative behaviors create positive social change, which then motivates more generative behavior, ultimately leading to a positive ripple effect. Had we solely relied on the quantitative results, we would have missed the important idea that mentoring engendered generativity consciousness! Additionally, we would have singularly dismissed generative behavior's contribution to socially responsible leadership, yet in the qualitative phase participants indicated that generative behavior and social change had a cyclical relationship.

In the Currie et al. (2009) examination of English secondary schools, their quantitative analysis revealed that none of the school contextual variables were clearly associated with the chosen leadership approach (results‐oriented vs. professional value‐based). My favorite analytic line in this article summarizes perfectly how explanatory sequential mixed methods designs can be helpful in evaluative practice: “We expected, therefore, that the reality was more complex than a simple one‐to‐one correspondence between the context and the leadership approaches. Our qualitative examination supported this hunch” (p. 673). People are complex! Who knew? The qualitative results indicated that the leadership approach was neither a purely rational choice nor a narrow choice based on the school's immediate context.

Analysis in exploratory sequential designs

Within an exploratory sequential mixed methods design, integration typically involves using the qualitative results to build the quantitative strand, namely utilizing the qualitative results to develop a measure, model, instrument, intervention, and/or product that will be tested in the quantitative phase. At the end, the researcher or leadership educator can demonstrate how the quantitative results built upon the qualitatively determined instrument, model, or materials.

Regarding the leadership transfer study highlighted above (Hastings et al., 2021), youth and adult community leaders interviewed in the qualitative phase were asked a series of eight questions related to their experiences with leadership transfer in their community. Grounded theory analyses revealed that the process of successful leadership transfer started with a small group of community leaders who did something—they helped pass a sales tax initiative for economic development, they started a leadership development program, they established a community philanthropic fund—something. That something created a contagion effect of hope within the community, which made it the “cool thing to do” to be involved in the community. So, what happens when more people in the community, on average, are engaged? More people to draw from for community leadership roles! Thus, an environment conducive for leadership transfer was created.

Our mixed methods task at this point was to develop a testable model from the grounded theory. The emergent model of leadership transfer led to a series of testable propositions using rural poll data, namely Belief in Community Leadership was hypothesized to predict Hope in Community, and Hope in Community was hypothesized to predict Civic Engagement. Structural equation modeling (SEM) results indicated that the hypothesized model fit the data adequately; however, several qualitative participants indicated they engaged in their community because they were asked. This qualitative result suggested perhaps both a direct and indirect effect of Belief in Community Leadership on Civic Engagement; thus, a mediation model was also tested (which meant we added a path from Belief in Community Leadership directly to Civic Engagement). SEM results indicated that the mediation model fit the data adequately, and a nested model comparison indicated that the mediation model was a better fit. But, contrary to expectation, Civic Engagement was negatively predicted by Belief in Community Leadership in the mediation model—whoa!

So, SEM results indicated that the mediation model better fit the data, where Belief in Community Leadership predicted Hope in Community, Hope in Community predicted Civic Engagement, but that Belief in Community Leadership, only when mediated through Hope in Community, had a positive impact on Civic Engagement. We realized that the mediation model perhaps painted a more complete picture of the qualitative results than the original hypothesized model, in that, the environment for effective leadership transfer (via broadened civic engagement) was facilitated when community hope became contagious based upon the community development efforts achieved by a small group of community leaders. Had we not tested the grounded theory model from the qualitative phase, we would have missed the critical function of hope contagion in leadership transfer! The qualitative phase helped us create much stronger hypotheses for testing.

Within LD practice, you could, for example, utilize the exploratory sequential mixed methods approach to create an intake assessment via interview and observation protocols to diagnose leadership and leader behaviors and the needs from a leadership intervention. From there, you can qualitatively analyze these diagnostic phase data to determine which leadership theories apply, feedback that diagnostic data with the client or organization, and share your analysis of applicable theories. The “Leadership Assessment” phase could then follow guidelines from Guthrie and King's (2004) Feedback‐Intensive Program as well as from Kroeck et al.’s (2004) guidance on leadership assessment. The key is to match technique and instruments with assessment needs: “…we argue that the theory in use should in large part explain the choice of assessment tools and techniques” (Kroeck et al., 2004, p. 74). In sum, utilizing exploratory sequential mixed methods presents you the opportunity to theoretically‐ground LD programs, which lead to sound program design and delivery decisions. When design and delivery of leadership programs fall prey to lacking theoretical foundations, evaluation and research efforts on such programs add a little contribution to the field.

PARTING COMMENTS

Advancing the field of leadership through rigorous empirical research and program assessment needs to involve advancing scholar and practitioner understanding of how to rigorously apply sound methodologies. Much of the “whale”‐sized complexity in leadership phenomena can be addressed through the use of mixed methods; however, if we just collect a multitude of various forms of data without sound methodological design and analytic practice, it is like eating the whale without any teeth—it might look like we are eating the whale, but in reality, we are just flapping our gums and not really getting anywhere. Thus, it is incumbent upon us as leadership researchers and/or program administrators to understand and articulate when a research problem or program evaluation effort is best served by mixed methods and how to maximize contribution through exemplary execution of quantitative and qualitative integration practices. Sound decision making in design, execution, and presentation of mixed methods studies and assessments allows us to “eat the whale one bite at a time” with teeth, not only producing better research and assessment outputs, but also greatly increasing the value of the results to scholarly, practitioner, and public audiences.

Biography

Lindsay J. Hastings serves as the Clifton Professor in Mentoring Research and Research Director for NHRI Leadership Mentoring at the University of Nebraska—Lincoln. She teaches undergraduate and graduate courses in leadership theory and leadership development, and her research centers around advancing scholar and practitioner knowledge in leadership mentoring, community leadership development, and youth leadership.

Hastings, L. J. (2022). Don't be afraid to eat the whole whale! Using mixed methods to enhance what we learn in leadership research and assessment. In Rosch D. M. & Hastings L. J. (Eds.). New Directions for Student Leadership: No. 175. Research and assessment methods for leadership development in practice, pp. 105–114. 10.1002/yd.20524