Theoretical and conceptual frameworks across local evaluation efforts in a nationwide consortium

Abstract

This paper describes the theoretical and conceptual frameworks used to guide the site-level evaluations of Building Infrastructure Leading to Diversity (BUILD) programs, part of the Diversity Program Consortium (DPC), funded by the National Institutes of Health. We aim to provide an understanding of which theories informed the evaluation work of the DPC and how the frameworks guiding BUILD site-level evaluations are conceptually aligned with one another and with the consortium-level evaluation.

INTRODUCTION

There is a strong argument for using evaluation theory to help inform how a particular evaluation is conducted. This is because evaluation theories are well-thought-out frameworks that are intended to help the evaluator make decisions about procedures and practices in support of the primary purpose of the evaluation. In other words, evaluation theories offer a logic for attaining a specific outcome for an evaluation. For example, if the purpose of the evaluation, as described by the theory being used to guide it, is to provide information to be used to direct immediate action by specific program decision makers, utilization-focused evaluation (Patton, 1997) would be a good choice of theory to guide procedural decisions. Thus, understanding the theories that were used to guide an evaluative study can help explain its intended aim and purpose, the procedural choices made, the kinds of information generated, and the intended uses of the findings.

In this chapter we describe the theoretical and conceptual models used to guide the program-level evaluation studies of the National Institutes of Health’s Diversity Program Consortium (DPC)¹ initiative called Building Infrastructure Leading to Diversity, or BUILD.² The purpose is to elucidate the “whys” of the site-level evaluation designs, procedures, and outcomes to provide further context for the studies and the knowledge produced by them. This is intended to promote a fuller understanding of the evaluation efforts of this large-scale national study.

The DPC was developed to better understand the outcomes of a variety of education and training experiences designed to enhance the diversity of the biomedical researcher work-force. BUILD programs were established at 10 institutions of higher education that serve geographically and racially diverse populations. The institutions include Historically Black Colleges and Universities, Hispanic-Serving Institutions, Asian American/Native American/Pacific Islander-Serving Institutions, and projects with targeted outreach to special populations (e.g., individuals in foster care). Each of the 10 programs was funded to contract with an evaluator to conduct a local-level study of its individual program. All BUILD sites were also required to participate in and support the DPC consortium-wide evaluation carried out by the Coordination and Evaluation Center (CEC) at the University of California, Los Angeles (see Chapter 1 by Guerrero et al. for a discussion of the complexity of this endeavor). Site-specific evaluation efforts examine program processes and outcomes that are unique to that BUILD program and outcomes (called Hallmarks of Success) that were identified as common across the BUILD programs. Some common data were used to inform both the local and consortium evaluations, while other data were collected by the CEC (often with support from the local-level evaluation teams) and used mainly for the consortium-level evaluation.

EVALUATION THEORY

An evaluation theory, also referred to as an evaluation approach or model, describes the purposes of and procedures for conducting an evaluation study. While prescriptive in nature, evaluation theories are not step-by-step, how-to guides because they are intended for use across various contexts. Thus, theories are best understood as providing a working logic and rationale for designing evaluations, including the procedures and methods used to operationalize specific outcomes for the evaluation, that is, utilization of evaluation findings (Christie & Lemire, 2019).

In locating the primary elements of evaluation theories, three dimensions emerge: methods, use, and the role of valuing and values in evaluation ((Alkin & Christie, 2023). Methods refers to the methodological techniques used in evaluation to promote it as activity that builds knowledge. Use describes the role of evaluation in improving programs, and thus is focused on how evaluation information is used, by whom, and for what purposes. The third dimension, valuing, concerns how value is placed on the program being studied.

Christie and Alkin (2023) contended that two perspectives inform the valuing process, one grounded in subjectivist and the other grounded in objectivist philosophical principles. The subjectivist approach argues that value judgments should be based on “subjective meaningfulness” because subjective factors govern all human behavior; an objectivist approach argues that value judgments should be made based on “publicly observable facts” (Diesing, 1966, p. 124). Several important ideas about the importance of traditionally underrepresented and marginalized groups in an evaluation have grown from subjectivist thinking and legal philosophies of justice, including deliberative democratic evaluation approaches (House & Howe, 2000), culturally responsive evaluation approaches (e.g., Hood, Hopson, & Kirkhart, 2015), and Indigenous evaluation approaches (e.g., LaFrance & Nichols, 2008).

We know from empirical studies (e.g., Christie, 2003; Coryn, Noakes, Westine, & Schröter, 2011; Miller & Campbell, 2006) that more than one theoretical approach may be integrated to guide a particular evaluation study. We also know that evaluation policies have an important role in narrowing the theories used in practice. Because the primary purpose of an evaluation policy is to provide guidance on how, when, and with what purpose evaluations are carried out in an organization, evaluation policy serves as a guiding mechanism for practice. The current chapter aims to discern what conceptual ideas were used to inform the local-level BUILD evaluations, including whether any specific policies were considered as the studies were designed.

Conceptual framing of the CEC consortium-wide evaluation

The purpose of the CEC consortium-wide evaluation is to provide information to the broader research training and mentoring communities about effective approaches and other outcomes. This goal is informed by federal evaluation policies that emphasize the use of experimental and quasi-experimental design to identify best practices within education (Mark, Cooksy, & Trochim, 2009). Reflecting this purpose and shaped by federal evaluation policy, the CEC evaluation is guided by method-focused theories. As such, we drew upon an approach called theory-driven evaluation (Chen, 1990). We also aimed to produce information for our NIH collaborators to inform decisions related to the direction of the DPC. Thus, we also drew on the principles of utilization evaluation (Patton, 1997) to generate useful information for the NIH and to a lesser extent for BUILD program decision makers.

The CEC was explicit about the theories we intended to use to guide our study in our grant proposal and in subsequent documents that we produced as part of our work. In line with theory-driven approaches, one of our first activities was to work with each BUILD site to develop a program logic model from which we developed an overarching theory of change for each of the primary domains of the BUILD initiative (i.e., student-, faculty-, and institutional-level outcomes). We also worked closely with NIH program directors to be sure that our work would help inform key discussions and decisions related to the DPC for high-level NIH officials. Given the purpose of the DPC, we discussed at length how we might incorporate culturally responsive theories. While we talked extensively about how the voices of program recipients might and should be centered in our evaluation work, culturally responsive evaluation theories did not strongly inform the overarching design of the CEC evaluation.

For the current study, we were interested in understanding which theories informed the BUILD site-level evaluations. Given the focus of the BUILD programs, we asked specifically whether and how culturally responsive theories and/or theories focused more broadly may have informed the design and implementation of these evaluations. Thus, the following questions guided our inquiry: What theories of evaluation guided local BUILD evaluators? How have theories been operationalized in BUILD evaluators’ work? Why are BUILD evaluators using these particular theories and frameworks?

METHOD

To address the study questions, we conducted semi-structured interviews with lead evaluators from the local BUILD sites. These interviews were guided by an interview protocol that each evaluator received in advance of the interview. While we were aware of the demands on BUILD evaluators’ time, the rich discussion and resulting qualitative data were key to understanding the theories and conceptual frameworks that have guided their work. We conducted approximately 60-min interviews with seven BUILD program evaluators. Applying the conceptual framework articulated by Christie and Alkin (2023), we identified the primary theoretical framework used to guide each of these local-level BUILD evaluation studies (i.e., use, methods, or valuing) as well as other conceptual principles that influenced it.

We began the analysis by writing summary memos following each interview. We also transcribed the interviews and analyzed them in an iterative process. We used several methods of qualitative coding throughout our analysis. For the first round of coding, we began with open and in vivo coding followed by several rounds of versus and concept coding (Saldaña, 2016). These first-cycle coding methods enabled us to identify and categorize the thematic and conceptual similarities and differences across interviews (Saldaña, 2016).

Following the first cycle of coding, we engaged in analytical comparisons between the emerging categories and our understanding of evaluation theories. We also considered the particular conceptual frameworks in light of guiding policies referenced by the evaluator participants. We then employed focused and theoretical coding to integrate and synthesize the emerging themes from the evaluators’ experiences. By using reiterative rounds of qualitative coding and analytical comparison we were able to draw clear findings that address the what, how, and why of the evaluators’ theories and frameworks and the related observable actions in their evaluation work at the various BUILD sites.

FINDINGS

The most salient finding from our interviews is that use-oriented theories guided the work of all of the interviewees. These theories of evaluation focus primarily on how an evaluation can produce knowledge to inform decision makers and facilitate organizational change, providing a conceptual understanding of a particular program within its unique context to the various program stakeholders (Christie & Alkin, 2023). That is, the site-level BUILD evaluations represented in this study were designed to facilitate the use of information for a select set of stakeholders. Each evaluator discussed their role in facilitating a better understanding of the program for stakeholders and producing knowledge to guide decision making and program and organizational change. One evaluator described the primary goal of the evaluation as providing “usable feedback to change the institution” and to “help stakeholders understand what needs to be changed so that the institution can support these students even beyond funding that cannot go on in perpetuity.”

When asked who the primary audience for their site-level evaluation was, BUILD evaluators provided nuanced responses, identifying program and university leadership, the NIH and DPC, and the students participating in the program. However, participants identified the local BUILD program leaders as the primary recipients of the evaluation information. In the words of one evaluator, “my first allegiance is to the program.” This interviewee described establishing trust with program and university leaders by providing usable knowledge and feedback about the program, thereby impacting decisions about how the BUILD program is “delivered” or “implemented.” All of the interviewees mentioned the importance of working closely with university principal investigators to ensure the evaluation would be responsive to both program development and implementation decisions and the evaluation requirements of the DPC and NIH.

Specific use-oriented evaluation theories

Within the broader category of use-oriented evaluation theories are prescriptive theories of evaluation, including utilization-focused evaluation (Patton, 1997) and participatory evaluation (Cousins & Earl, 1995), among others (Christie & Alkin, 2023). The key components of interviewees’ work aligned most closely with the major phases of utilization-focused evaluation frameworks: development of users’ commitment to the intended focus of the evaluation; users’ involvement in determining methods, design, and measurement; users’ engagement in interpreting findings and making judgments; and users’ decision making about further dissemination (Christie & Alkin, 2023). BUILD evaluators who used this theory also spoke of identifying and engaging the primary intended users of the evaluation and, by fostering their involvement and buy-in, establishing a management-consulting role. As one evaluator noted:

People in academia are not as familiar with program implementation and management, and this means that for the first 2 years of academic programs, it seems like I am just working on organizational structures, how to administer programs, how to get stakeholder feedback, and implementing those in some way, shape, or form.

Two interviewees described using a participatory evaluation framework (Cousins & Earl, 1995), which involved both the active inclusion and capacity building of evaluation personnel and social researchers on the project along with the practice-based decision makers of the program and university. One evaluator created a work group that included a range of people responsible for the program and evaluation. The group met at least once a month, with other subgroups meeting more frequently.

Two interviewees also identified students as the primary stakeholders of the evaluation and articulated the importance of “leveraging students to change institutions.” One noted that “we need the people who we are serving (through the local BUILD program) to say, ‘this is what we see, this is what we need.’” For these evaluators, use-oriented evaluation should focus on highlighting the voices of the program participants—in this case, students—as the primary knowledge and essential feedback source for program development and implementation. But they also saw students as stakeholders, and they shared information about how the program was designed to assist them and about what the program was working toward—in other words, the goals and outcomes of the program. While both of these evaluators also highlighted the BUILD program and university leaders as key stakeholders, there was an important nuance in their perspectives: By characterizing students as vital stakeholders who should be aware of the program’s goals and processes of development, these evaluators incorporated student advocacy as a component of evaluation use.

Secondary theoretical approaches

In line with previous research (Christie, Quiñones, & Fierro, 2014), our interviewees drew upon conceptual principles associated with more than one theoretical framework. Thus, we also describe the “secondary” or less salient models used to guide the site-level evaluations of BUILD. The secondary models used were from the methods category, followed by valuing evaluation theories.

Methods theories

For evaluators whose secondary theoretical approach was methods focused, there was an emphasis on identifying a theory of change and then using experimental and quasi-experimental designs to assess the effects of implementing the theory of change (Chen & Rossi, 1983). In our conversations with BUILD evaluators, all but one spoke at length about developing a theory of change; a few saw the process of developing a theory of change with the program and university leaders as a key component of their work.

BUILD evaluators recognized the arguments for using experimental designs to measure program outcomes. At the same time, experimental designs were described as “insufficient” for measuring the true impact of local BUILD programs; this is aligned with an argument Chen and Rossi (1983) offered in theory-driven evaluation. Evaluators expressed a desire for evaluation policies that would have allowed them to better measure the potential effects of the program in relation to its goals as formulated by program leaders, rather than initiative’s Hallmarks of Success, which were frequently referred to as “just a list of things” or “a laundry list” that lacked an actionable program theory. One evaluator shared that while the Hallmarks provided “guidance for us” they also “could be taking time and energy away from measuring something else, something we want to do at a local level” In sum, BUILD evaluators saw the need for the formulation of a particularized and context-specific theory of change by local BUILD site leaders, followed by thoughtful and rigorous measurements of its implementation.

Valuing theories

BUILD evaluators also drew upon valuing theory, which was mostly aligned with Stake’s responsive evaluation framework and House’s social justice evaluation (Christie & Alkin, 2023). Both approaches place value on the perspectives of program recipients and amplify voices that are less emphasized in our society (Christie & Alkin, 2023). In alignment with Stake’s theory—which advocates for using thick description to highlight the voices of primary stakeholders who are invited to participate in a responsive evaluation—one evaluator discussed how their interactions with stakeholders influenced how they conducted the evaluation. After meeting with students and getting their perspectives on the program and ongoing evaluation, the evaluation team realized

[that] oral histories are used a lot of times with underrepresented populations. For events that really affect a person, so we thought that would be much better than just doing a semi-structured interview….We could ask them to tell their stories and understand how their identities have impacted their education and their life choices.

Another interviewee shared a similar experience: Student feedback initiated critical conversations within the evaluation team, which then led to a responsive adjustment in the evaluation.

Some BUILD evaluators showed strong alignment with House’s prescriptive theory of evaluation (House, 2023), which centers the less-acknowledged perspectives and voices within the program. One interviewee shared that BUILD students “are coming from a different community than the community I came from, and yet I’m working with an initiative that’s supposed to promote their interest. Well then, let’s promote their interest!” For three of our interviewees, underrepresented students were referred to as the primary stakeholders and were repeatedly emphasized in our discussions about whom the evaluation was for. One participant said, “I always think of all the evaluations I do as somewhat biased toward the primary stakeholder input. [I try] to involve them as much as possible without burdening them.” This sentiment was evident in our conversations with multiple evaluators and often led to discussions about theories of culturally responsive evaluations and social justice frameworks within the BUILD evaluation. We discuss this next.

Culturally responsive evaluation theories

We asked each interviewee how, if at all, they saw culturally responsive evaluation theories or theories of social justice guiding and/or influencing their BUILD evaluation work. Despite having already spoken about clear examples of culturally responsive or social-justice-infused evaluation activities, most could not identify an explicit theory or framework. One evaluator responded, “Not really, because this is what a good evaluator should already be doing.” Another said, “Not really. It’s inherent.” Yet another said, “Nothing explicitly, but it is inherent in our evaluation plans and questions.”

As noted in the section above on valuing theory, these evaluators tended to place great emphasis on including the voices and perspectives of those who are heard less in evaluation work. Recall the above discussion on use-oriented evaluation theories that saw students as the most vital stakeholders. For some evaluators, running a useful evaluation included providing students with the necessary knowledge to advocate to the program leaders for their needs. In one evaluator participant’s words:

I’ve come to believe in my evaluation work, whoever is being served by a program are usually pretty good lenses on the world and on the program. And they often know what they need or want. And being responsive to that within the context of the initiative…that is culturally responsive theory to me…highlighting their brilliant insights.

Another way we saw the evidence of culturally responsive and social justice theories of evaluation in action was through BUILD evaluators’ conscientiousness about the demands of both the national stakeholders and evaluation policies on the lives of the program participants. This was especially highlighted in their discussion of data collection: the frequency, quantities, types, and use of it. One evaluator said:

I think about burden on students, data collection, burden of evaluation data collection on students. We collect a lot of data from them, we ask them for a lot…and I feel conflicted: I understand this is a research project, but I feel like there is a fine line between supporting these students and using them as research subjects. I think this is an equity issue that needs to be talked about.

Another evaluator described survey fatigue and its impact on diverse and underrepresented students. They commented that it seemed like the people who are meant to be helped and positively impacted are negatively impacted by the plethora of data collection resulting from what several evaluators characterized as a policy tension between academic research expectations and the local evaluation.

IMPLICATIONS

While not surprising, it was encouraging to find that the interviewees used theory to inform the design and implementation of the site-level evaluation studies. It was also important to learn that the evaluation studies across the BUILD sites that participated in this study are aligned with one another theoretically and with the consortium-level CEC evaluation of the DPC. Indeed, we saw an across-the-board adoption of use-oriented evaluation theories by site-level BUILD evaluators. The most widely used secondary theory was theory-driven evaluation, which was incorporated within a user framework. This is theoretically aligned with the CEC’s utilization-focused evaluation and theory-driven evaluation models to guide and inform the DPC-level evaluation.

Evaluators implemented theories in their evaluation activities by: (a) identifying key stakeholders; (b) working to produce accurate and helpful information to guide stakeholders in their decision making; (c) building capacity and evaluation “buy-in” at their sites; (d) including the voice of program recipients in their study designs; and (e) developing theories of change. We observed evaluation products designed to promote use through an informal review of evaluation artifacts, such as infographics, executive summaries, PowerPoint presentations, websites, and regular meetings with program users. These kinds of reporting strategies have been shown to increase the usefulness of evaluation information and to promote learning (e.g., Torres, Preskill, & Piontek, 2005).

By approaching an evaluation study with use-oriented frameworks, BUILD evaluators distinguished between the various stakeholder groups and included program leadership and university leaders, the DPC, and the NIH. BUILD site-level evaluators then identified the information and decision-making needs of each stakeholder/user group. For example, they distinguished between program leaders—who want rich information about how the program is functioning and whether or not it is reaching its site-specific goals—and the CEC, which needs data on each of the Hallmarks of Success to produce peer-reviewed manuscripts and information to inform federal policy and future investment in programs. Given the range of intended uses for evaluative information, the BUILD evaluators have creatively and diligently worked to provide useful and relevant information for all intended users. In practice, interviewees described conducting two concurrent evaluations—one at the site level and another at the DPC level—to increase the usability of the information generated by the evaluation and to help ensure that the evaluation studies would be meaningful and relevant. (See Chapter 7 in this issue for a discussion of national–local evaluation partnerships.)

Given the commitment to and focus on supporting the consortium-level evaluation, BUILD site-level evaluators developed theories of change that incorporated the consortium-level evaluation’s indicators and outcomes. Developing theories of change at each site was important because it helped the evaluators gain clarity from the BUILD principal investigators on the goals of the local-level program outcomes, and later about how the activities were intended to lead to key program outcomes. In some cases, interviewees described how the theories of change helped to focus site-level evaluation questions. These practices and their rationale reflect the principles of theory-driven evaluation frameworks.

With respect to the use of culturally responsive and justice-oriented theories, interviewees did not report explicit or intentional integration of these frameworks into their study procedures. However, as they described their practices and the issues they were considering while planning and implementing each stage of the evaluation, we found clear evidence of the presence of these theoretical ideas, especially when they explained the rationale behind some of their procedural choices. In this way, these theories described the theoretical positions the evaluators had taken as advocates for social change. For example, one evaluator said they believed we should be elevating student voices to ensure that the next cohort of students would be engaged in a program that is more sensitive to the unique complexities and opportunities of serving students who have been systematically excluded from the biomedical research workforce. This might be expected given the focus on the BUILD program and the timing of our interviews, which were conducted after the murder of George Floyd and in the context of today’s racial justice movement. Moreover, it did seem to be the case that these evaluators were drawn to working with their programs because of the focus of the BUILD initiative.

Thus, it may be that the theories that inform an evaluator’s position as an advocate should be better understood and that we should better recognize and discern the role of the evaluator as an advocate for social change and for a particular program under study that might result in positive program outcomes. In the case of the BUILD evaluators we interviewed, we found evidence to suggest they were uniformly advocates for pushing against structural and institutional inequities and in seeing their work as a mechanism for advancing change. As a result, they implemented strategies and procedures that they thought would help to achieve this outcome.

Biographies

AUTHOR BIOGRAPHIES

Dr. Christina Christie is professor and Wasserman Dean of the School of Education and Information Studies at UCLA and serves as multiple principal investigator for the Coordination and Evaluation Center and director of the Evaluation Core for the Diversity Program Consortium.

Carmel Wright earned her M.Ed in Social Research Methodology from the School of Education and Information Studies at UCLA and is currently enrolled in the Law School at the University of California, Davis.