Five ways to get a grip on the shortcomings of logic models in program evaluation

Abstract

Logic models are perhaps the most widely used tools in program evaluation work. They provide reasonably straightforward, visual illustrations of plausible links between program activities and outcomes. Consequently, they are employed frequently in stakeholder engagement, communication, and evaluation project planning. However, their relative simplicity comes with multiple drawbacks that can compromise the integrity of evaluation studies. In this Black Ice article, we outline key considerations and provide practical strategies that can help those engaged in evaluation work to identify and mitigate some limitations of logic models

For decades, logic models have been a quintessential tool for program evaluation. They provide relatively simple, diagrammatic representations of the ‘how’ and ‘why’ that underlies how specified interventions function.^1,2 Often, they illustrate the strategic intentions behind interventions, which highlight the perceived linkages between planned activities, and anticipated outcomes³. Logic models are frequently used to guide communication with program stakeholders^1,4 as well as to identify target areas for monitoring and evaluation.⁵ However, given several limitations, some question whether they are an appropriate tool for current evaluation intitiaves.^6,7

Black Ice

As commonly used, logic models focus on succinct, visual synthesis of program components, and linearly depict program functioning. Correspondingly they can lead stakeholders to:

1. Ignore the complexity that underlies social and educational interventions⁶. This can manifest, for example, through inattention to variable contextual factors (e.g., resourcing, political pressures) as well as to cultural and value differences that may compound to influence intervention outcomes.

2. Assume that a given program is a legitimate solution to an identified problem; this can constrain thinking about external influences on the problem or limit exploration of adaptations to enhance contextual compatibility.^2,8 For example, a serious oversight might involve an intervention to enhance youth self-esteem that does not acknowledge or attend to underlying systemic oppression of racialized groups.

3. Neglect to identify the undesirable outcomes that interventions can inadvertently precipitate.⁹ Even well-intended interventions can result in problematic effects.¹⁰ Though unintended these effects are not necessarily unpredictable.^11,12 Rather, adverse outcomes may be anticipated by purposeful attention to research evidence or experiential expertise.^11,13 Examples of such effects include evidence that cultural competency training can in fact trigger expressions of racism toward cultural minority groups¹⁴ or that global health educational innovations can increase trainees’ willingness to perform skills outside their scope of training.¹⁵

Overall, these limitations make some evaluators hesitant to use logic models.⁶ There are, however, useful strategies that can minimize these limitations. Here, we draw attention to five ways to get a grip on these drawbacks of logic models while reaping the benefits they offer.

1. Articulate negative logic

Proactive work to identify how interventions may lead to undesirable outcomes or harm during logic modelling has been described as negative or dark logic modelling.⁹ Supplementing a conventional logic model with one that outlines the potential downsides of an innovation (e.g., loss of autonomy and identity; challenging one’s self-efficacy) has several advantages. It allows for harm mitigation planning, while informing evaluation designs such that they proactively monitor undesirable and adverse outcomes. Further, it promotes sharing of balanced information about both risks and benefits associated with innovations.

2. Perform a logic analysis

A logic analysis provides a deeper level of scrutiny of select claims that are presented in logic models.^16,17 A direct logic analysis is a formative process that examines whether the program design and implementation is consistent with available evidence about the critical conditions required for achieving desired effects.¹⁶ A reverse logic analysis is a summative process that contrasts program characteristics with alternative models of realizing similar outcomes. Evidence for logic analyses can be found by synthesizing research evidence or soliciting experiential expertise.¹⁸ Whereas both forms of logic analysis offer valuable checks of the credibility and trustworthiness of the claims posited in logic models,¹⁹ they require substantial effort¹⁶. Thus, we recommend judicious selection of specific sub-sets of the logic model for logic analysis such as areas of recurrent implementation difficulties, key program principles, and/or values.¹⁶

3. Complement traditional logic models with more sophisticated designs

There have been numerous evolutions in evaluators’ approaches to visually illustrating the ‘hows and whys’ of intervention functioning. Thus, traditional logic models can be meaningfully complemented with more sophisticated approaches. For example, nested logic models incorporate multiple logic models that are linked through shared activities or outcomes and thus ‘nested’ into one another.²⁰ They compartmentalize the structural complexity of multi-program systems: the highest level model of a nested logic model appears concise, but is constituted by multiple, aligned ‘sub-logic models.’^2,20

Action models are another contemporary option that are more expansive than traditional logic models. In addition to articulating pathways via which a program may influence specified outcomes, action models outline the prescriptive assumptions about how those pathways may be activated.²¹ Specifically, they include details about implementation protocols, required characteristics of target participants and implementing organizations, essential capacities of front line implementers, as well as critical norms or resources in the program’s ecological context (i.e., culture, norms, and resources of the greater system, such as an academic institution or health services unit).²¹ Action models can be optimal tools for guiding evaluations focused on implementation processes. Recent scholarship outlines a typology of logic models that includes a relatively advanced type four logic model. Type four logic models both 1) outline the change mechanisms of interventions (in lieu of listing discrete program activities and resources), and 2) illustrate the contextual factors upon which target outcomes depend.⁶ The result is a more flexible type of logic model that allows for variation in activities and outcomes across contexts.⁶

4. Be clear about the purpose of the logic model

The rationale for using logic models should inform decisions about how to employ the technique.⁶ For example, if the primary goal is to engage stakeholders and build consensus about program activities the simpler, conventional model may be optimal. The aim here is to capture stakeholder attention by being both aesthetically appealing and visually efficient¹ (i.e., maximizing accurate interpretation while minimizing cognitive effort²²). Where evaluation of implementation is being prioritized, action models may be optimal, due to their expansive details on the prescriptions for program functioning. More complex models (e.g., Type 4 models) are not ideal for stakeholder engagement or consensus building ⁶. Instead, they are instrumental for guiding reflections about how to improve the intervention’s use in organizational context.⁶ INotably, they can be a valuable toolkit for evaluation-leads, who wish to improve or optimize program monitoring and evaluation designs.

5. Cyclically review and update logic models

Typical logic models often represent perceived rather than verified relationships among program activities and outcomes. Further, illustrated relationships may change due to evolving contexts and outcomes as programs are implemented.⁷ As such, logic models should be treated as dynamic rather than static, with an expectation that they will evolve over time.⁷ We recommend cyclical efforts to revisit and revise logic models as contexts changes (e.g., COVID-19 pandemic) and as evaluation data become available.

Conclusion

Logic models offer a pragmatic approach to synthesizing information about interventions that could otherwise be challenging to describe. Clarity about why a logic model is needed is critical to deciding how to go about creating one. The five strategies outlined above can help educators and evaluators get a grip on the limitations of logic models and maximize their utility.

Conflicts of Interest

We have no conflicts of interest to declare.

Funding

This project was not funded.