Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2023 Jun 16.
Published in final edited form as: New Dir Eval. 2022 Aug 8;2022(174):21–31. doi: 10.1002/ev.20507

Implementing case study design to evaluate diverse institutions and STEM education contexts: Lessons and key areas for systematic study

Krystle P Cobian 1, Damani Khary White-Lewis 2, Sylvia Hurtado 1, Hector V Ramos 1
PMCID: PMC10275575  NIHMSID: NIHMS1903799  PMID: 37333467

Abstract

This chapter highlights the important contributions of case study research to the evaluation of student-centered programs and broader STEM initiatives in higher education. We summarize the Diversity Program Consortium’s case study evaluation of the Building Infrastructure Leading to Diversity (BUILD) initiative, funded by the National Institutes of Health (NIH), with aims to enhance diversity in the NIH-funded workforce. We describe lessons learned from the case study design used for the evaluation of BUILD that applies to administrators of STEM initiatives who are interested in case study methods and to evaluators who are familiar with case studies and tasked with program evaluation of a multisite STEM program. These lessons include practical considerations for logistics and the importance of clarifying the goals of the case study design within the larger program evaluation, fostering the continuation of knowledge within the evaluation team, and embedding trust building and collaboration throughout all stages of the case study.

INTRODUCTION

Enhancing diversity, equity, and inclusion in science, technology, engineering, and mathematics (STEM) disciplines in U.S. higher education has been a prolonged project spanning several decades (Tsui, 2007). Often, federal agencies, private foundations, and program evaluators aim to understand the short- and long-term impacts of STEM initiatives on the individual career success of women and racial groups historically excluded from the scientific workforce (Felix, Hertle, Conley, Washington, & Bruns, 2004). More recently, there is added interest in studying how such initiatives create sustainable institutional-wide change (Reeves, Bobrownicki, Bauer, & Graham, 2020). This complex, multilevel approach characterizes recent efforts by the National Institutes of Health (NIH), the National Science Foundation, and the Howard Hughes Medical Institute, which now provide resources for evaluation of large-scale intervention programs designed to holistically support groups that have historically been excluded from science education and training.

One example of a large-scale investment in postsecondary STEM education and its evaluation is the Diversity Program Consortium (DPC) (n.d.). The DPC is a network of institutions and programs, funded by the NIH, to improve scientific training and mentoring and enhance diversity of participation in biomedical research careers (McCreath et al., 2017). Diverse and primarily undergraduate teaching institutions across the United States received Building Infrastructure Leading to Diversity (BUILD) awards, which provided funding for the implementation and evaluation of a range of programs and structural changes aimed at engaging and increasing the retention of individuals from diverse backgrounds in biomedical research, and ultimately, the biomedical workforce. The 10 BUILD sites were funded for student training, faculty development, and institutional capacity-building activities (Hurtado, White-Lewis, & Norris, 2017).

While each BUILD site conducts local program evaluation for summative feedback and reports to the NIH (see Chapter 7 by Hwalek et al. for more detail), the Coordination and Evaluation Center (CEC) was funded to conduct a longitudinal, consortium-wide evaluation of the training and mentoring interventions that BUILD and National Research Mentoring Network (NRMN)1 awardees develop and implement. (Guerrero et al. discuss the complexity of this endeavor in Chapter 1.) The DPC developed process and outcome indicators at critical training and career-transition points called Hallmarks of Success (McCreath et al., 2017). As described in greater detail, the Hallmarks span student-level, faculty-level, and institutional-level impacts (DPC, n.d.). While the CEC administers and manages consortium-wide surveys and survey data collection in coordination with each BUILD site, it also conducts case studies as part of the overall consortium-wide evaluation (Moses et al., 2020).

Case study research is used to understand a phenomenon within its context (Stake, 2011. Due to the importance of understanding how programs operate within their unique context, this approach is utilized often in several fields of study and in program evaluation (Yin, 1992, 2017). Case study research is becoming more widely accepted in health services research (Crowe et al., 2011) and in the science and engineering fields more broadly. For example, case studies can explain various processes for achieving sustainability and institutionalization of program components and innovations within their unique context (Cobian & Ramos, 2021). This is critical considering the diversity of higher education institutions with respect to geographic locations, state education policy environments, student demographics, and institutional missions and aspirations.

Case studies aim to provide multiple forms of evidence to lend insight into the implementation of BUILD awards and help to explain the processes that lead to observed outcomes (Davidson et al., 2017; McCreath et al., 2017). The utility of the case study method is the holistic approach to understanding processes in their context (Stake, 2011), which can address a number of the institutional-level outcomes identified in the Hallmarks of Success.

This chapter aims to describe how case study research can effectively contribute to the evaluation of large-scale, multilevel, multisite, undergraduate programs and broader STEM initiatives. We provide a brief summary of the case studies developed for the DPC evaluation, focusing on the approach, design, implementation, and analysis processes. (For additional details on the qualitative analysis of the case studies, see Moses et al. [2020].) We also provide lessons learned from the ongoing case study project conducted by the CEC to evaluate the BUILD initiative. This chapter will show how case study design can be implemented to improve understanding of institutional systems change, particularly for initiatives aimed at supporting diversity.

IMPLEMENTING CASE STUDY DESIGN FOR THE DPC EVALUATION

Case study research utilizes multiple forms of evidence. Still, it is a predominantly qualitative method in which a bounded entity, program, or system (a “case”) is studied at length in its real-life context (Stake, 2011; Yin, 2017). Case studies are useful when the boundaries between the phenomenon of interest and its larger context are not clearly distinguishable and cannot be manipulated, as is typically the case with experimental methods (i.e., conducting multiple experiments within a controlled environment; Yin, 2017). Multiple case studies involve the study of several cases that are often replications of the first case, and therefore may share similar features. When multiple cases—such as BUILD sites—are examined, findings are generated by comparing the cases to fundamentally understand the quintain, or the umbrella of cases representing the phenomenon of interest. Evaluating the large-scale, multisite BUILD initiative required an in-depth understanding of each site’s program and its unique institutional context to understand the broad phenomenon of interest–aspects of the programs and/or institutions that were driving program impacts (Moses et al., 2020).

The primary evaluation question of interest for the DPC evaluation is: How are BUILD programs building capacity and infrastructure for primary and partner institutions to advance diverse student success in biomedical research training? (For additional questions, see Moses et al. [2020].) BUILD awards were granted for 5 years, with an opportunity for an additional 5-year renewal. Considering the potential length of time, the programs could operate on each campus, the case study evaluation team planned to conduct two phases of site visits. The first phase was an exploratory multiple case study (Yin, 2017) to understand how and what the campuses chose to implement (beyond their descriptions in the funded proposals), and why they took on various forms despite a common starting point in the NIH request for proposals. The second phase of case studies will be an explanatory multiple case study (Yin, 2017) to understand, given this extensive knowledge about the BUILD sites and initiatives, the processes and challenges for program impact and institutionalization as funding begins to sunset.

Data collection

In case study research, qualitative and quantitative data are drawn from multiple sources in order to inform understanding of each case (Yin, 1992). To obtain a contextual understanding of each site prior to physical site visits, the research team compiled data from the Integrated Postsecondary Education Data System (IPEDS) and reviewed campus websites in order to develop campus profiles. The team also conducted individual interviews, focus groups, and observations of BUILD activities and collected documents, including reports and articles shared by participants.

Identification of key participants for interviews and focus groups was important for ensuring that we collected data from individuals who could provide insight into the BUILD initiative’s impact at each campus. Case study research team members used their knowledge of higher education and STEM education contexts to identify participants with different vantage points about the BUILD initiative. Additionally, the case study research team conducted pre-site visit interviews virtually with the BUILD principal investigators and central BUILD faculty and staff members in order to gain an understanding of the program components and the launch of the program at each campus.

During each site visit, members of the research team conducted interviews with additional faculty and administrators at the campus to get a sense of program impact and initial outcomes. At the conclusion of each visit, the researchers held an in-person meeting with the BUILD team to provide an initial briefing about the research team’s observations and to exchange questions about key issues that the case study team sought to better understand. In many cases, the final meeting confirmed observations and the case study research team was able to offer suggestions to meet the DPC’s goals. Finally, in order to ensure that the case study team obtained the perspectives of all individuals relevant to understanding BUILD programs at each site, the team conducted virtual post-visit interviews with individuals who participants recommended.

It is important to note that BUILD site stakeholders collaborated with the case study research team throughout the process of data collection and initial analysis. For example, the case study team members worked with local site liaisons to determine participants and ensure that the sites stayed informed about the case study visits. The case study team also held debriefing meetings with each site’s core BUILD program leaders. These meetings allowed for further insight and exchange between data collectors and program administrations and served as a form of member checking (Lincoln & Guba, 1986).

Analyzing the data

In multiple case study design, the creation of narrative reports allows researchers to begin to understand the phenomenon of interest within each case’s context. The case study team developed a narrative report for each BUILD site to accomplish this. Each narrative report began to weave the case study team’s analysis, including publicly available data, participants’ voices, relevant STEM education theories, and the DPC Hallmarks, to provide preliminary assertions that began to answer the case study evaluation questions. Members of the case study evaluation team wrote reports, transcribed interviews and focus groups, and coded at the student, faculty, and institutional level, and corresponding Hallmarks of Success. (For additional details, see Moses et al. [2020].) Each level had common but also some unique codes to capture participants’ vantage points within the interventions and across the larger institutional context. Members of the team conducted cross-case analysis—an in-depth exploration of similarities and differences within and across BUILD sites—in order to test assertions using matrices (Miles, Huberman, & Saldaña, 2014). For example, a cross-case analysis can be conducted to examine experiences of faculty in culturally aware mentorship training within and across sites to understand the efficacy of the intervention and how faculty experience challenges or support for intentions to encourage students to pursue biomedical careers (White-Lewis, Romero, Gutzwa, & Hurtado, 2022).

Qualitative analysis can be particularly challenging in a multiyear study when case study team membership shifts because this can lead to loss of institutional knowledge and insight from data collection. For example, in the case study team, membership shifted between the data collection and data analysis phases. While these shifts posed a challenge to maintaining contextual knowledge of the BUILD sites, incoming team members “visited” sites by reading through all campus narrative reports, interviews, and collected data, and met regularly with the case study team to discuss and make sense of the data. Additionally, some members of the case study team have been involved since the beginning and have been present at most of the site visits. Regular discussions during analysis and writing were most helpful in raising questions and reexamining conclusions.

We continue to analyze the case study data from the first phase in order to address the Hallmarks of Success; these data are also combined with survey data from the larger DPC evaluation in order to answer DPC evaluation questions in multiple- or mixed-methods studies. Data from the first exploratory phase of site visits also informed areas to explore in more depth for the second phase of site visits, which aims to explain BUILD program outcomes.

LESSONS FROM THE DPC CASE STUDY

Next, we describe important lessons learned from the case study design and implementation to evaluate the BUILD initiative. The lessons are ordered based on when they occurred during the process of design and implementation. All of the lessons incorporate practical and theoretical considerations.

Anticipate and adapt to logistical challenges

Given the scale of the current BUILD case studies, we provide some practical guidance for evaluators to anticipate the logistics and resources necessary for an effective single or multiple case study of a STEM program evaluation. Our multiple case study consisted of 10 site visits to institutions across the United States. Our team included approximately four to eight researchers at any given time who collected interview and focus group data from approximately 500 students, faculty, program administrators, evaluators, and senior institutional administrators. Additionally, the COVID-19 pandemic impacted plans for data collection between the first and second phases of site visits. With scale and COVID-19 in mind, we share a few considerations for future evaluations.

First, robust demographic information was essential both for annual reporting to the funders (the NIH) and for analysis, given that the program was focused on understanding what interventions work for underrepresented groups as defined by the NIH. As such, one practical recommendation is to make sure to request that participants complete demographic data forms to gather information about their social identities. Second, institutional review board changes pertaining to consent and COVID-19 suggest more online interaction, meaning that participants must receive materials earlier and verbal consent is recorded as part of interviews. Third, we recommend developing a well-organized process for cloud-based protective storage of all case study data so that analysts can easily access documents while working remotely. For example, our team used Dedoose to code transcripts, which allowed several team members to continue coding and analysis while working in different locations and on different devices. The team also transitioned work-flow and processes to ensure all analytic memos and documents were digital and could be shared securely online. Secure “box” sites were used for storing confidential information, whereas team writing and analysis was facilitated with Google docs and spreadsheets.

Another set of recommendations pertains to recruitment and local site logistics. To build participant recruitment across multiple sites, each BUILD site appointed a knowledgeable contact who could facilitate identification of participants among faculty, staff, students, and campus administrators–the main actors engaged with BUILD. Site contacts attended regular site-based meetings (held on Zoom) that focused on implementation and local evaluation to become familiar with the case study team leads. These BUILD site contacts also provided information on relevant partners both within and outside of the campus who had insight into the site’s initiatives. In some cases, site contacts were helpful in scheduling interviews and rooms for in-person interviews.

For any site contact and site principal investigators, we found it important to provide a detailed set of requirements to clarify needs for identifying participants for the study. This step was critical for enhancing consistency in data collection efforts. The consequence of not providing detailed guidance to partners enlisted to help with identifying case study participants could lead to over-interviewing of individuals who may not be able to answer the study’s research questions, or to bias from either the case study team or the site contact in identifying participants. Site contacts and principal investigators were instrumental in providing feedback about who to interview in order to answer the case study team’s research questions. It is also helpful to have a detailed organization plan, including strategies for (1) tracking progress with data collection for each site, (2) managing and protecting participant data, (3) coordinating interview schedules and travel, (4) curating notes and memos from data collection, and (5) training team members on coding data to ensure reliability of the analysis. Doing so ensures that progress across multiple sites is visible and can be shared across the data collection team.

Clarify the goals of the case study design within the larger program evaluation

A strong case study design requires clear short-term and long-term goals and routine re-examination of these goals in order to ensure an effective program evaluation. Indeed, much like the logic models that sites used as guides for their own short-term and long-term goals, evaluation efforts may also have phases during a larger program evaluation wherein specific evaluation tasks are more appropriate. For the BUILD initiative, the goal in the first phase was to better understand implementation of the strategies that campuses used to fulfill the aims articulated in the Request for Proposals (RFA) of the grant-funded initiative; in the second phase, we will focus on institutionalization and program impact. However, the case study team also understood there was a clear short-term need to obtain a multifaceted understanding of how programs were being implemented on each campus while the larger DPC evaluation began collecting survey data to measure attitudes, beliefs, and behaviors that contributed to program outcomes at the student and faculty level.

Some case study designs lean toward being theoretically driven from the beginning of the project, which allows for more precision and efficiency with data collection and analysis. This can narrow the focus of what to examine, which can make the process of identifying assertions and themes that answer the evaluation questions easier (Yin, 1992). An explanatory study has more focus and is especially relevant to augmenting quantitative results from each site. An exploratory study, on the other hand, allows more room for an inductive analysis to generate emergent themes but can be broad and unwieldy, especially with multiple team members involved and a large amount of data.

Finding a balance between both approaches is necessary: A purely inductive approach to understanding STEM initiatives is not possible considering the large body of research that exists and influences current STEM education and training programs. Additionally, a theoretically driven case study is not always feasible in a complex, multisite program where each site employs different theories to guide local implementation efforts. Moreover, several STEM theories and literature guided the development of the Hallmarks of Success because they needed to span all sites. Thus, common theories could be referenced in relation to specific outcomes and commonly implemented initiatives.

Finally, considering the multitude of decisions involved in case study design, the over-arching program evaluation approach should also be considered to ensure that the case study aligns with the particular paradigm, lens, or epistemological stance of the evaluation. For example, Mertens and Hopson (2006) described several evaluation models sensitive to diversity, culture, and power. Likewise, Boyce (2017) considered a STEM program evaluation approach that explicitly sought to examine how an education program is equitable. She shared the challenges and opportunities that arise as evaluators continue to expand on efforts and strategies for melding equity-driven evaluation approaches with STEM programs that aim to broaden participation for historically excluded groups. The present study employed an evaluation model that was sensitive to issues of power and equity (see Chapter 5 by Maccalla et al.); however, future program evaluations might consider an explicit equity-driven evaluation design and a subsequent case study design that extends this work.

Employ a range of methods to explore outcomes

Examination of program outcomes can be challenging when working to evaluate a large, multi-site program with several levels of potential impact. One solution is to employ a wider variety of data collection and research methods in order to answer key evaluation questions. Specifically with the case study team and the broader CEC evaluation, the CEC identified opportunities to mix survey data with case study data in a variety of mixed methods designs (Creswell, 2014) in order to answer important research questions that could otherwise not be answered as thoroughly by only relying on one form of data. For example, the case study team examined biomedical bachelor’s degree rates at each BUILD site to identify the extent to which degree production increased in each major by race and gender. This information could then be supplemented with interviews from site participants who had more insight and the ability to explain increases or decreases in certain disciplines. Researchers on the comprehensive evaluation team can work together on mixed methods design (Creswell, 2014) where quantitative and qualitative data are mixed and analyzed in an intentional order and with intentional priority given to one form of data over the other, in order to answer the research question(s) driving the next stages of the Enhance Diversity study. In other words, case study researcher can collect and use multiple forms of data in order to systematically investigate the impact of an intervention. Additionally, case study data can be combined with survey data from the evaluation in order to answer other research questions where the survey data provides key measures from site participants in aggregate form, and case study data provides details about how and why there may be differences in across sites on key variables.

Another suggestion that entails using multiple methods to explore outcomes involves returning to case study data collected in the exploratory phase and supplementing with additional collection of data to understand change (only possible in multiyear studies). For example, in the first phase of case study site visits, the research team found that efforts to increase biomedical research publications varied considerably among BUILD sites for both students and faculty. This led the team to return to examine this pattern by collecting information about BUILD-funded research—as indicated in peer-reviewed publications and grant numbers found in several biomedical databases—in order to further document and measure the impact of BUILD funding on the production of publications on each campus. Doing so allowed researchers to examine program impact and diffusion of BUILD funding, specifically by recording the amount of biomedical research articles published by faculty and students at each institution, involving BUILD-affiliated trainees and mentors. This process was facilitated by supplementing the original data collection with an additional step of collecting publicly available information after the first phase of case study data collection.

Finally, causal mapping can be a useful approach to account for outcomes (Ackermann & Alexander, 2016; Miles et al., 2014). Qualitative data are organized into “maps” that link outcomes to processes and decisions made by individuals within the organization. In turn, these linkages create a visualization of what actions likely caused the next set of actions that led to an outcome. Another related analytical and presentation method is program mapping, which helps compare overall initiatives across multiple sites as well as in-depth intervention program elements within a single site (Reeves et al., 2020). A single program map can be simple or complex; it can identify program areas (e.g., faculty mentor development, curricular reform, summer program enrichment) where greater institutionalization has occurred and other program areas that are likely to sunset after grant funding ends based on conclusions drawn from interviews and site visits. It is important to note that this is facilitated by obtaining tracking data of participants in each the program areas, information which is also useful to campuses for reporting to funding agencies.

Foster the continuation of knowledge within the evaluation team

Evaluation teams, particularly for large-scale programs, will inevitably have turnover in membership. For example, as we noted above, the case study team’s membership shifted between data collection and data analysis for the Phase 1. Good case study research requires a continuation of knowledge with respect to the rich details and memories of the analysts. As such, case study teams should plan for turnover in membership by developing systems to onboard new analysts and ensure that research artifacts, decision-points, and case knowledge are shared with team members and threaded throughout the case study.

The team can create an organization plan for developing and storing (in a secure case study database) its analytic memos, audit trails, cross-case matrices, and all analytic notes used to make sense of the data. All artifacts and data collected from each case or site should be organized in the database. These multiple forms of evidence all contribute to informing analysts about the phenomena of interest. This calls for greater transparency in analytical steps and saving work that demonstrates the team’s decisions. The case study team members should also maintain and regularly review artifacts to retain memory of cases and contexts.

Incoming case study team members should be provided with an orientation that describes the organization system and key documents in order to reduce the overwhelming amount of data and increase team members’ capacity to support data collection and analysis. The orientation can guide them through the evaluation goals, overarching questions and subquestions, the current phase of the case study, and an overview of the cases. Likewise, an exit process for outgoing team members can help with retention of contextual knowledge that may be held by anyone leaving the project. The process can be as simple as writing one final analytic memo that details aspects of the case study design where the case study team member has unique knowledge or experience. This process can also involve establishment of a mutual agreement that an individual will stay on as a co-author for a developing manuscript or be available for consultation for the duration of the case study data collection, analysis, and dissemination of findings.

Embed collaboration and trust building throughout all stages of the case study

Stakeholder participation and collaboration are critical for a successful case study because they ensure trustworthiness and can lead to richer collection of data. This was especially evident considering the multiple sites involved and the intended collaborative agreement between all institutions in the DPC. Mutual collaboration and consideration of stakeholders and participants is especially important considering the DPC’s aims of supporting students from underrepresented groups and the institutions that serve them.

It is important to note that collaboration was structured into the DPC evaluation through an agreement among the NIH and all grantees involved. This immediately set the expectation that we would work together and that participants would have a way to provide input into the cross-campus evaluation. Other evaluations that do not have formal collaborative agreements may have to make additional efforts to collaborate with stakeholders and obtain buy-in at each site.

Moreover, conversations and site visits in the first phase allowed the CEC to establish the evaluation team as engaged and committed to the same goals as each of the campuses: improving pathways into biomedical careers and educating the next generation of scientists. This was accomplished through thoughtful observations, recognition of challenges and successes, and immediate feedback that sometimes confirmed their own assessments about the progress of their efforts. For some, it helped highlight the forest instead of the trees, reminding them of the value of their day-to-day of their work improving STEM training for underrepresented groups.

CONCLUSION

As funding initiatives shift toward large-scale investments aimed at institutional transformation, internal and external evaluators must also shift approaches in order to conduct assessments that monitor program goals, modify approaches, and provide external feedback (McCreath et al., 2017) in ways that address the complexity of organizational change. Additionally, efforts to evaluate diversity-focused programs will also need to be evaluated with more attention to embedding culturally responsive and equity-based practices at each step of the evaluation process. Program evaluation efforts utilizing case study design can provide a rich, in-depth understanding of program implementation and effectiveness and provide opportunities for collaboration, reciprocity, and equity in collecting and analyzing data. This understanding can supplement quantitative measures in mixed methods and/or multiple methods evaluation in order to increase understanding of program outcomes, or it can stand alone in presenting multifaceted dimensions of program implementation.

When evaluating a program aimed at enhancing diversity, it is critical to consider ways that evaluation efforts can mitigate harm to the underrepresented groups central to the program itself. The DPC evaluation prioritizes stakeholder and participant collaboration (Davidson et al., 2017), and a case study has the capacity to further advance those goals through the design and implementation of data collection and analysis. As a method that has been utilized for program evaluation—one that is becoming more familiar to scholars in science disciplines—there is much that can be learned from prior case study evaluations to inform the field of evaluation studies and the work of evaluators of STEM initiatives.

A well-designed case study can contribute a rich understanding to the evaluation of an initiative or funded program. In this chapter, we have highlighted how case study design can effectively contribute to the evaluation of large-scale, multilevel, multisite undergraduate programs and broader STEM initiatives. We described lessons derived from the multiple case study design we implemented to understand the BUILD initiative as part of the larger DPC evaluation. The lessons we shared in this chapter are particularly relevant to evaluators of student-centered programs and broader STEM initiatives. As funding agencies shift toward initiatives aimed at more complex institutional and systemic change in STEM training and career development, case studies can help capture the complexity of processes that connect to program outcomes.

Biographies

Krystle P. Cobian, PhD, is a research analyst at the Diversity Program Consortium’s Coordination and Evaluation Center at UCLA.

Damani Khary White-Lewis, PhD, is an assistant professor of education at the University of Pennsylvania whose research focuses on faculty hiring, promotion & tenure, mentoring, and retention in STEM disciplines.

Sylvia Hurtado, PhD, is a professor of education at the UCLA School of Education and Information Studies whose specialty is diversity in higher education and STEM interventions.

Hector V. Ramos, PhD, is a research analyst at the Diversity Program Consortium’s Coordination and Evaluation Center at UCLA.

Footnotes

1

The NRMN is a national network of mentors and mentees from all biomedical disciplines relevant to the NIH mission, with a variety of programs that provide mentorship, professional development, mentor/mentee training and networking opportunities to individuals from the undergraduate to early career faculty levels. NRMN Phase II began in fall of 2019 to continue to develop mentoring and networking opportunities for biomedical researchers from diverse backgrounds, from undergraduates through early career faculty.

REFERENCES

  1. Ackermann F, & Alexander J (2016). Researching complex projects: Using causal mapping to take a systems perspective. International Journal of Project Management, 34(6), 891–901. 10.1016/j.ijproman.2016.04.001 [DOI] [Google Scholar]
  2. Boyce AS (2017). Lessons learned using a values-engaged approach to attend to culture, diversity, and equity in a STEM program evaluation. Evaluation and Program Planning, 64, 33–43. 10.1016/j.evalprogplan.2017.05.018 [DOI] [PubMed] [Google Scholar]
  3. Cobian KP, & Ramos HV (2021). A cross-case analysis of developing program sustainability and institutionalization in early stages of a multisite biomedical student diversity initiative. BMC Medical Education, 21, Article 254. 10.1186/s12909-021-02663-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Creswell JW (2014). Research design: Qualitative, quantitative, and mixed methods approaches. SAGE Publications. [Google Scholar]
  5. Crowe S, Cresswell K, Robertson A, Huby G, Avery A, & Sheikh A (2011). The case study approach. BMC Medical Research Methodology, 11(1), 100. 10.1186/1471-2288-11-100 [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Davidson PL, Maccalla NM, Afifi AA, Guerrero L, Nakazono TT, Zhong S, & Wallace SP (2017). A participatory approach to evaluating a national training and institutional change initiative: The BUILD longitudinal evaluation. BMC Proceedings, 11(Suppl. 12), 157–169. 10.1186/s12919-017-0082-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Diversity Program Consortium. (n.d.) New DPC Hallmarks, Years 6–10. Retrieved from: https://www.diversityprogramconsortium.org/pages/hallmarks_yr6-10
  8. Felix DA, Hertle MD, Conley JG, Washington LB, & Bruns PJ (2004). Assessing precollege science education outreach initiatives: A funder’s perspective. Cell Biology Education, 3(3), 189–195. 10.1187/cbe.03-10-0017 [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Hurtado S, White-Lewis D, & Norris K (2017). Advancing inclusive science and systemic change: The convergence of national aims and institutional goals in implementing and assessing biomedical science training. BMC Proceedings, 11(Suppl. 12), 17. 10.1186/s12919-017-0086-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Lincoln YS, & Guba EG (1986). But is it rigorous? Trustworthiness and authenticity in naturalistic evaluation. New Directions for Program Evaluation, 30, 73–84. [Google Scholar]
  11. McCreath HE, Norris KC, Calderón NE, Purnell DL, Maccalla N, & Seeman TE (2017). Evaluating efforts to diversify the biomedical workforce: The role and function of the Coordination and Evaluation Center of the Diversity Program Consortium. BMC Proceedings, 11(Suppl. 12), Article 27. 10.1186/s12919-017-0087-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Mertens DM, & Hopson RK (2006). Advancing evaluation of STEM efforts through attention to diversity and culture. New Directions for Evaluation, 109, 35–51. 10.1002/ev.177 [DOI] [Google Scholar]
  13. Miles M, Huberman AM, & Saldaña J (2014). Qualitative data analysis: A methods sourcebook (3rd ed.). Sage Publications. [Google Scholar]
  14. Moses MW II, Romero A, Gutzwa J, Ramos H, MacCalla NMG, Purnell P, & Hurtado S (2020). BUILD program evaluation: Case study analysis methods [Technical report]. Diversity Program Consortium Coordination and Evaluation Center. Retrieved from: https://www.diversityprogramconsortium.org/briefs/pages/BUILD_evaluation [Google Scholar]
  15. Reeves PM, Bobrownicki A, Bauer M, & Graham MJ (2020). Communicating complex STEM program evaluation to diverse stakeholders. CBE—Life Sciences Education, 19(2), es4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Stake RE (2011). Multiple case study analysis. Guilford Press. [Google Scholar]
  17. Tsui L (2007). Effective strategies to increase diversity in STEM fields: A review of the research literature. The Journal of Negro Education, 76(4), 555–581. Retrieved from: http://www.jstor.org/stable/40037228 [Google Scholar]
  18. White-Lewis D, Romero A, Gutzwa J, & Hurtado S (2022). “Moving the science forward”: Faculty perceptions of culturally diverse mentor training benefits, challenges, and support. CBE—Life Sciences Education, 21(1), ar2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Yin RK (1992). The case study method as a tool for doing evaluation. Current Sociology, 40(1), 121–137. 10.1177/001139292040001009 [DOI] [Google Scholar]
  20. Yin RK (2017). Case study research and applications: Design and methods. Sage publications. [Google Scholar]

RESOURCES