Policy over- and under-design: an information quality perspective

Abstract

This article seeks to improve our understanding of what policy over- and under-design mean; what are the consequences of these suboptimal designs; and how politics matters to these designs. Based on the review of the literature and a variety of examples that focus on the role of information quality in policy design, and drawing on two phenomena from the field of epidemiology (namely, over-adjustment and unnecessary adjustment), the article enhances the definitional clarity of the terms under investigation and allows us to address the difficulty in reconciling technical problem solving with politics. The article proposes new definitional statements of proportionate and disproportionate policy designs that vary according to the extent to which the main design properties are adjusted to low-quality information. It also explores distinct variations in a few policy characteristics resulting from over- and under-design. The policy characteristics examined here include economic efficiency, policy effectiveness, policy robustness; the space for making significant changes at a later stage when high-quality information becomes available; the potential consequences in terms of policy over- and under-reaction; and the political ramifications of these suboptimal designs for elected executives. With regard to the interaction between technical and political logics, the article posits that deliberate policy over- and under-design can be viewed as solutions to serious political problems faced by elected executives, especially in politically sensitive times. Therefore, it facilitates a more realistic understanding of the conditions under which a policy is under- or over-designed to respond to some objectives but entirely proportionate with regard to others.

Recurring complaints regarding many public policy decisions and processes cite their complexity, costliness, time-consuming and cumbersome nature, on the one hand, or their ambiguity, inconsistency, political flexibility and their vulnerability to capture, on the other. The analytical thread undergirding these complaints is comprised of two concepts, policy over-design and policy under-design, which were introduced into the policy lexicon largely as part of the recent explosion of scholarly interest in policy design (Haynes and Li 1993; Howlett 2018, 2019a; Howlett and Kemmerling 2017; Howlett et al. 2015; Nair and Howlett 2016, 2017). A single attempt to clarify their meaning has produced the following definitions: under-design occurs when suboptimal designs are adopted, while over-design, by contrast, results when “the reverse occurs and resources are wasted in ‘over engineering’ a design vis à vis the significance of the problem” (Howlett and Kemmerling 2016, 5). So far, however, no attempt has been made to systematically define these terms, delve into the consequences of policy over- and under-design, or address the difficulty involved in reconciling an analysis of policy design with political analysis.

To fill this gap, this conceptual article draws on existing literature and a variety of examples in order to shed new light on these constructs, emphasizing the central role that the quality of information plays in how policy is designed. Information refers here to statistical and qualitative evidence, as well as the beliefs that motivate professionals and mobilize the public (Baumgartner and Jones 2015). It is a crucial asset for policy designers as well as for ‘design-for-policy’ entrepreneurs (Lewis et al. 2020) in their daily work, and it is, therefore, the key to many policies and their improvement. Information quality refers to information on costs, timescales, benefits and risks that is accurate, timely, contextually appropriate for the task, clearly represented and accessible to policy designers. To capture the complexity of policy design, the article diverges from the standard policy theory assumption, according to which government efforts are ‘seamlessly calibrated’ to the policy problem at hand (e.g., Howlett 2019b). It identifies two terms—over-adjustment and unnecessary adjustment—which are drawn from epidemiology, wherein they are used with regard to controlling for relevant covariates when drawing inferences about causation based on observational data regarding exposure to harm (e.g., maternal smoking) and outcome (e.g., neonatal mortality). These terms are employed here in order to advance the definitional clarity of our core constructs, and to address directly the difficulty involved in reconciling technical problem solving with politics.

The article proposes new definitional statements of proportionate and disproportionate policy designs that vary according to the extent to which the main design properties are adjusted to low-quality information. It also explores distinct variations in a few policy characteristics resulting from over- and under-design. The policy characteristics examined here include economic efficiency, policy effectiveness, policy robustness (Capano and Woo 2017); the space for making significant changes at a later stage of the policy design process when high-quality information becomes available; the potential consequences in terms of policy over- and underreaction; and the political ramifications of these suboptimal designs for elected executives. Regarding how politics matters to policy over- and under-design, this article posits that policy over- and under-design can be seen as solutions to serious political problems faced by elected executives, especially in politically sensitive times. This is because deliberate over- and under-design of a technical problem may solve a political problem by inflicting damage on political rivals and/or successfully shaping voters’ perceptions in a favorable manner. Therefore, the article facilitates a more realistic understanding of the conditions under which a policy is under- or over-designed to respond to some objectives but entirely proportionate (or correct) with regard to others.

The article is structured as follows. The first section discusses the role that information plays in the policy design process; the second defines the concept of proportionate policy design; the third subsequently introduces the terms over-adjustment and unnecessary adjustment, in addition to defining the concept of policy over-design; while the fourth section delineates the concept of policy under-design. The fifth section discusses the policy characteristics and the political implications of disproportionate policy design. The sixth section elaborates on how technical problem solving and politics interact during policy design processes, and the seventh section proposes relevant methodologies. We close our article with a discussion of avenues for future research.

Information in the policy design process

Policy design is “the process of creating a policy response to a policy problem” (Peters 2018, 1). In terms of policy instruments, it refers to “the deliberate and conscious attempt to define policy goals and to connect them to instruments or tools expected to realize those objections” (Howlett et al. 2015, 292; see also Majone 1975; May 2003; Howlett 2019a). Within policy studies, policy design has been linked to research addressing policy implementation and policy instruments (May 2003), policy ideas and policy formulation (e.g., Linder and Peters 1990a, b; 1991; James and Jorgensen 2009), and governance (Howlett 2009, 2014; Howlett and Rayner 2007). Those who subscribe to the manner of thinking advocated by political science, among them the authors of this article, regard policy design as involving (1) politics; (2) conflicts—concerning the values pursued and the evidence that can be used—which originate from differences among policy paradigms (Hall 1993) and from variances in the social and political construction of both the problem and the target population (Schneider and Sidneu 2009); and (3) the engagement of social actors and citizens (Peters 2018).

Policy over- and under-design can arise due to a multitude of factors, among them lack of effective feedback (Dryzek 1983; May 1991, 1992); under-specification of design mechanisms (Schneider and Ingram 1988, 1990; see also Majone 1980); misreading evidence from policy evaluation (e.g., Elmore 1980, 1987); ignoring spillovers, sleeper effects and unintended consequences of policy design (e.g., Dunn 1997; Lindblom 1990); and radical uncertainty and wicked problems (Daviter 2017, 2019; Head 2019; see also Nair and Howlett 2017; Howlett 2019a). The fact that humans are “disproportionate information processors” (Jones 2001, p. 23; see also Jones and Baumgartner 2005) can likewise lead to policy over- and under-design. We also make information the primary focus of this article, directing attention to variations among policy designs with regard to their quality of information (Stone 2002, 22).

Therefore, we deviate from the standard policy theory assumption, according to which government efforts are ‘seamlessly calibrated’ to yield efficient and effective policy solutions to policy problems (Howlett 2019a), and from the view that policy design and re-design are “strongly committed towards the ‘linear’ achievement of an established goal (end-state)” (Capano 2009, 11). This divergence from the standard assumption has gained solid support among scholars (e.g., Simon 1973; de Leon 1999; Hargrove 1975; Rittel and Weber 1973; Walker et al. 2001, 2010; Kwakkel et al. 2010; Howlett 2019a; Mathijssen et al. 2007; Daviter 2019). Studies have also demonstrated that policy actors interpret scientifically validated information according to their belief-system and mental models (e.g., Sarewitz 2004; Weible et al. 2008; Kahan 2016; Heikkila et al. 2020); that there are contextual frictions in the conversion of this information into policy (e.g., Bennett and Howlett 1992; Jasanoff 1987; Radaelli 1995; see also Daviter 2015); and that when science truly leads policy making, nationally established ideas may place policies on trajectories that ignore countervailing new information until the latter becomes overwhelming (Baekkeskov 2016). Note that, some policies emerge from a process that is not designed (Howlett and Mukherjee 2014), and therefore may involve no relevant evidence, and that ideological designs are characterized by the absence of real evidence (Peters 2018, 50).

In this article, information refers to statistical and qualitative evidence, as well as the beliefs that motivate professionals and mobilize the public. We adopt this broad definition because “[policy] problems are in part subjective and in part objective, and any policy solution to a problem will have clear distributional effects” (Baumgartner and Jones 2015, 7). Information quality is defined as information on costs, timescales, benefits and risks that is accurate, timely, contextually appropriate for the task, clearly represented and accessible to policy designers. It can be measured, for example, by using a questionnaire comprised of 16 dimensions, namely accessibility; the appropriate amount of information; believability; completeness; concise representation; consistent representation; ease of manipulation; freedom from error; interpretability; objectivity; relevancy; reputation; security; timeliness; understandability; and value-added (Strong 2009). Operational definition of these dimensions could be found elsewhere (Wang and Strong 1996; Najjar and Schniederjans 2006). Information quality includes therefore both objective dimensions (e.g., accuracy) as well as subjective dimensions that are defined relative to the context of a policy designer performing a certain task which requires the information (e.g., timeliness) (e.g., Strong 2009; English 2001; Mai 2013; Gustavsson and Wänström 2009).

Low-quality information includes, for example, random information; unvalidated scientific information; subjective information created by the designers rather than provided by multi-actor networks of public and private stakeholders; information that reflects a disproportionate influence of a scientific view when scientific views are wide-ranging and dependent on a scientist’s field of expertise (e.g., the failure to fully consider the perspectives of experts beyond epidemiology during the fight against the first wave of COVID-19 in Israel); and misleading statistics produced throughout the policy design process. Misleading statistics may originate, for example, from errors in assumptions. Specifically, over-design may incur additional costs when actual conditions are less severe than assumed conditions, while additional failures may result from under-design when actual conditions are more severe than the assumed design conditions. Another example is a situation involving the mistaken assumption that a certain capacity of a policy-related element will follow the normal distribution in the event of a flood, when, in fact, it follows a gamma distribution, thereby leading to over-design. Alternatively, statistical errors may occur in sampling. The ineffectiveness of a sample affects trend detection: positive and negative autocorrelation lead to overestimation and underestimation, respectively, of a trend. Errors in trend detection can lead to either over-design or under-design, thereby misleading policy designers (Pathak et al. 2017). In addition to errors in assumptions and sampling, data gaps in big data pertaining, for example, to minority groups (Giest and Samuels 2020), can furthermore undermine the quality of information available to policy designers.

The effect of low-quality information in policy design is demonstrated, for example, by a case in which such information supports incorporating the fixed value of redundancy in all policy instruments in a policy design mix. This may lead to under-design or over-design because not all policy instruments are vulnerable to error to the same degree, and policy instruments may not require the same level of protection against errors or against unexpected day-to-day events (e.g., absence of employees due to sickness; power outage). Furthermore, many policy instruments exhibit a high degree of inherent error resilience, and not all policy instruments in a policy design mix exert the same degree of influence on the final policy outcome.

Information is therefore a crucial asset for policy designers in their daily work, whether the policy is simple or complicated (i.e., multidimensional). Not surprisingly, information selection lies at the core of decision-making in the policy design process (Simon 1973). A major obstacle in executing this task is information overload (e.g., Edmunds and Morris 2000; Jansen 2008), which may be strategically induced (Persson 2018) by a variety of policy entrepreneurs (e.g., Maor 2017a) and other actors. Studies have also discussed information avoidance (Case et al. 2005) or filtering (Miller 1960) by decision-makers. In political science, Walgrave and Dejaeghere (2017) demonstrated that politicians partially outsource their information selection to procedures and/or staff members, personally apply rigorous rules of thumb concerning what they should attend to and what they should not, and compensate for the pressure and constant risk of error with a large dose of self-confidence. Given the limited capacity of the human brain and the fact that much information is noise (Gigerenzer and Todd 1999), focusing on the variance in the quality of information available to policy designers during a policy design process can give the terms under investigation an analytical value. Attention first turns to the definition of proportionate policy design.

Proportionate policy design

The (dis)proportionality of “matching instruments to goals, policy problems, social impact and organization” (Linder and Peters 1990a, 307) is deduced by evaluating the quality of information used during three stages. The first stage comprises the policy modeling; the second, the formulation of implementation and governance approaches (e.g., evaluation, control); and the third, the exploration of opportunities for fine-tuning and calibration of policy tools (Salamon 1989, 2002). This evaluation is performed while taking into account the jurisdiction’s unique context (e.g., economic conditions, cultural background), as well as the unique characteristics of target audiences.

Proportionate policy design refers to an intermediate or final policy design the main properties of which are adjusted to high-quality information, thereby increasing the probability that the power of the policy instrument/s supports the policy’s performance. High-quality information (e.g., scientifically validated information; strong comparators) can be gathered when the policy problem has been effectively articulated and well posed in the first place; when the proposed design is based on an informed theory; and when policy goals are relatively clear. The abundance of information of varying quality requires policy designers to separate the wheat from the chaff, that is, to gather the information necessary to formulate stable policy rules and to allow for some degree of freedom to adjust policy incentives over time in order to increase social welfare. Yet it also requires that policy designers ignore some information, preferably low-quality, unnecessary information. The study of heuristics—efficient cognitive processes that ignore information—indeed shows that less information, computation and time can improve accuracy (Gigerenzer and Brighton 2009). Proportionate policy design occurs, at times, when policy designers employ heuristics and ignore low-quality information.

Among other things, the aforementioned definition implies that high-quality information should be gathered in order for policy designers to accurately ascertain the intensity of the constraints within which the intended policy will be implemented and in light of which the policy design process will be undertaken. This process of ascertaining the intensity of the constraints within which the intended policy will be implemented has two facets, which vary along the temporal dimension. In cases of short-term policy goals, the aim of policy designers is to accurately gauge the intensity of current constraints and fully incorporate this information into the design process at the proper time. In cases of long-term policy goals, the aim of policy designers is to ascertain (1) which constraints cannot be modified; (2) which constraints can be ignored; and (3) which can be modified (and how) before implementation begins in order to increase the chances of policy success (e.g., by changing attitudes, ideas and norms). Thereafter, this information should be fully incorporated into the design process at the proper time. The same rationale may be applied to relationships among policy problem elements—the substantive nature of the policy problem (Peters 2005; Peters and Hoornbeek 2017)—if those are viewed as implicit or explicit forms of constraints. Gathering high-quality information regarding external and/or internal constraints is vital if policy designers need to accommodate abstract information or any information which is temporarily indistinguishable from random data (e.g., when the definitions of the policy problem and the viability of policy solutions are not stable across time; Rice and Fineman 2004).

The implementation period of a proportionately designed policy is optimal: it follows all implementation milestones as planned; it meets its intended goals; and it does more good than harm when taking into account significant impacts. An example of a proportionate design for a complex policy problem and a complex policy solution is air traffic regulation. An example of a proportionate design for a simple problem and a simple solution is automobile speeding regulation (Howlett 2019a, 17). Another example is the design of the national electricity transmission systems in France and the UK in the middle of the twentieth century, which superimposed the national priorities upon the regional and local. This resulted in well-designed and efficient systems, in comparison with many other European countries in which local municipalities were allowed to maintain control, resulting in weak networks (Helm 2014). An additional example in the area of public bureaucracy is the design of the U.S. Food and Drug Administration’s drug approval policy, which at times relies on federal advisory committees. Drugs submitted to the FDA for review between 1989 and 2000 that underwent review by an advisory committee before approval were significantly less likely to encounter safety problems after they were released into the market (e.g., drug warnings and withdrawal from the market) than those which did not receive an advisory committee review, all else being equal (Moffitt 2014). Whether a particular policy design constitutes an instance of over- or under-design will be relative to proportionate policy design.

Policy over-design

Policy over-design refers to an intermediate or final policy design the main properties of which are adjusted to high-quality information and, in addition, to lesser- or low-quality information, thereby increasing the probability that some of the power of the policy instrument/s does not contribute to the policy’s performance. An example of policy over-design for a simple policy problem and a simple solution are the tightly specified military interventions in the area of national security (Howlett 2019a, 17). A related example is the dropping by an Israeli F-16 fighter jet of a one-ton laser-guided bomb into a densely populated neighborhood in Gaza on 22 July 2002, killing Hamas military leader, as well 14 other people, among them nine children.

Given that policy designers have at their disposal high-quality information, when (and under what conditions) are they likely to be influenced by low-quality information? Low-quality information may derive from selective bias and information overload. It may also derive from relying on a static view of a policy problem, leading ultimately to the over-design of policy solutions that are characterized by slow adaptation to new conditions (e.g., Harou et al. 2009). However, low-quality information may also be produced throughout the policy design process. At times, policy designers rely on excessive number of variables; wish to achieve an excessive level of precision; and/or exercise poor statistical judgment when considering the effect of interest, the method used to estimate that effect, and the control variables considered in the analysis.

Regarding excessive number of variables, policy design is an attempt to uncover the expected causal effects between policy instruments and the policy outcome of interest. However, many effects on the policy designers’ menu can be examined, ranging from total effects (i.e., direct and indirect effects), direct effects, indirect effects, joint effects, conditional effects and others. Excessive number of variables or parameters, uninformed by substantive knowledge, can (1) obscure a true effect of key factors of interest, (2) create an apparent effect when none exists and (3) generate a biased effect, which may direct policy designers to select the no-action route. This surplus information may also lead to unnecessary adjustments among component parts of the policy design and between them and the outcome of interest.

Regarding the level of precision, an excess of precision “may [lead to] narrowness that is inappropriate with problems that are themselves broad and complex” (Peters 2018, 91). Excessively specific policy design may occur, for example, when policy designers are constrained by too many rules, procedures, and absolute standards; when policy designers seek to match a particular instrument to a particular problem in a relatively mechanistic manner (i.e., according more weight to complicated technical instructions over simple ones); when there is political motivation to target (social) policies via cumbersome eligibility criteria so that money is not “wasted” on non-eligible citizens; and when politicians design overly complex policy schemes that are easier to capture (Helm 2010) in order to obtain the support of organized interests (Lerner 2009; Aalbers et al. 2013; Lehmann and Söderholm 2018).

Regarding the presence of low-quality statistical information in the policy design process, poor statistical judgments includes, for example, overfitting, over-adjustment and unnecessary adjustment. Overfitting occurs when the statistical analysis undertaken by policy designers corresponds too closely or exactly to a particular set of data, thereby failing to fit additional data or predict future observations reliably. Policy designers may consequently find themselves relying on statistical models characterized by poor generalization ability. Over-adjustment occurs when an adjustment is undertaken for intermediate variables even though the total causal effect of policy instruments on policy outcome is of interest. Such an adjustment will often bias estimates of the total causal effect toward null (Schisterman et al. 2009). This point is derived from epidemiology research (Schisterman et al. 2009; VanderWeele 2009) and can be applied to the sub-field of policy design because the development of methods and theory concerning causality lies at the hearts of both research areas.

Whereas the term over-adjustment might be used for any setting of policy design in which control for a variable introduces (rather than eliminates) bias, the term unnecessary adjustment might be used for any setting in which control for a variable does not affect bias but may affect the precision of the policy design (Schisterman et al. 2009). Unnecessary adjustment during the policy design stage is a phenomenon that occurs when policy designers’ control for a variable does not affect the expectations regarding the estimate of the total causal effect between a policy instrument and a policy outcome (adapted from Schisterman et al. 2009). Unnecessary adjustment in cases of technical problem-solving may occur, among others, in three cases: (1) adjusting for a completely irrelevant variable; (2) adjusting for a variable that causes solely negligible (policy) outcomes; and (3) adjusting for a variable whose only causal association with variables of interest is limited because it is not in the causal pathway identified. The result of adjustment for such variables is that the total causal effect of a policy tool on policy outcome remains unchanged (in expectation). Some unnecessary adjustment of variables could harm designers’ estimates by decreasing the precision of the policy design. This, in turn, entails the risk of undermining policy reliability. However, when combined with high-quality information it increases the probability that some of the power of the policy instruments will not be used to support the policy’s performance.

Importantly, in cases involving the interaction of technical and political logics, over-adjustment (in its common usage rather than its statistical meaning) may occur, for example, when political executives face policy problems that can only be solved by granting concessions to rival political actors who hold radically different opinions. An example is Theresa May’s Brexit deal, which was designed following negotiations with, and subsequently concessions to, competing party factions, namely the pro-Brexit Democratic Unionist Party and the European Research Group, and to two groups of former Remainers—those in favor of a soft Brexit and those who support a second referendum. This Brexit deal was rejected by the U.K. parliament.

Adjusting the main properties of the policy design to high-quality information—alongside low-quality information—has two-fold consequences for the policy design product. First, the intermediate/final policy design becomes more robust (i.e., characterized by overcapacity to maintain the functions of policy despite uncertainty) or complicated than is necessary for achieving the intended policy goal. Increased complexity facilitates regulatory capture, for example, as politicians incorporate loopholes so that the policy only simulates effectiveness (as in the case of EU emissions Trading Schemes, see Helm 2010, 195). Second, the intermediate/final policy design becomes less flexible, limiting the possibility of making changes at a later stage of the policy design process, when high-quality information becomes available. This low degree of flexibility may be exacerbated when decision-makers embed in the policy design low frequency of policy reviews, explicit adjustment rules, and when they establish an independent entity that will make policy adjustment decisions (Gawel and Lehmann 2019).

In addition, over-design may waste resources due to the unnecessary extra quality of the policy—which entails extra costs, time, effort and material—alongside the design risk incurred and the design opportunities forgone. Cumbersome and complex policy designs also make policy implementation difficult, furthermore contributing to rising implementation costs. The implementation duration of over-designed policy might be either shortened because of the earlier introduction of low-quality information (when no correction is undertaken at a later stage) or extended due to the added effort and/or because some of the power of the micro-elements in a policy mix design are only partly used, or not used at all. Policy over-design can be avoided by delaying some decisions to later stages, when decision-makers will have a clear picture regarding which parts of the available information are unnecessary or wrong and, by contrast, which are necessary and, therefore, should be acquired.

Policy under-design

Policy under-design refers to an intermediate or final policy design the main properties of which are adjusted to lesser- or low-quality information, thereby increasing the probability that the power of the policy instrument/s will be insufficient to support the policy’s performance. An example of policy under-design for a complex policy problem and a simple solution is climate change (Howlett 2019a, 17). A further example is the design of the U.K.’s renewables policy, which lacks strict regulations aimed at minimizing party-political incentives that shape both the picking of winners and the mechanism that reflects capture (Helm 2010, 183–4). In fact, “[i]t is hard to think of a policy more open to capture—and an example where the capture has been more successful” (Helm 2010, 191). An additional example is the EU’s Common Agricultural Policy, one of the EU’s most economically inefficient policies, which lack strict regulations that are designed to minimize rent-seeking, capture, and lobbying by landowners and producers (Helm 2017).

Policy under-design may result, among others, from earlier design decisions that were made on the basis of little high-quality information, due to time constraints; high-quality information that was either underestimated or accidentally lost during the earlier design stage; or high-quality information that was introduced too early into the policy design process for its true value to be fully appreciated. Policy under-design may also occur when policy designers wish to save design costs; when policy problems are inherently ambiguous; when the objects of their designs tend not to behave in predictable ways (the latter three reasons may also lead to over-design); when the policy problem is too complex for a single policy (i.e., when design is excessively ambitious); and when decision-makers’ staff and advisors are characterized by low technical capacity. An example of earlier design decisions based on meager high-quality information is an emotion- or empathy-based policy design (e.g., disaster relief dispatched to developing countries), which stands in contrast to evidence-based policy design.

Consequently, the intermediate/final policy design becomes (1) less robust and/or too simplistic than necessary for achieving the intended policy goal, and (2) more flexible, creating opportunities for policy capture. This high degree of flexibility may be exacerbated if decision-makers define only general objectives and criteria for policy tool deployment, set rules for regular policy reviews, and establish a political entity that will make policy adjustment decisions (Gawel and Lehmann 2019). Note that, a relatively high level of flexibility in the design product allows for the ongoing and immediate incorporation of new information into the policy design.

Under-designed policies are prone to policy failures. Take, for example, the following technical problem-solving illustration. In the case of sustainable drainage for the key road network, the need to mitigate floods is paramount. Consequently, optimal design parameters are vital when designing the pavement structure. Insufficient high-quality information regarding the drainage performance of pavement structure, for instance, risks both under-designed drainage systems that fail too frequently and the possibility of groundwater contamination and flooding. Furthermore, when considering the challenges for water infrastructure planning and design brought about by climate change, under-designed solutions may not only fail frequently, but these failures may be magnified by the uncertainty in the long-term frequency and magnitude of future extreme events (Huia et al. 2018).

Under-designed policies may also result in additional policy maintenance or policy “repair” costs, and they may also be associated with increased expected costs due to higher anticipated failure costs. The implementation duration of an under-designed policy may be shortened (when no correction is undertaken at a later stage) because of the effort saved. Thus, although policy under-design is prone to failure, it entails lower design costs, a shortened implementation period, and leaves more room for maneuver in the future.

Policy characteristics and the political implications of policy over- and under-design

Both under- and over-designed policy proposals may result in sub-optimal policy outcomes when such policies are implemented ‘to the letter.’ Once an over-designed policy is implemented as planned, and social costs begin to mount up without producing offsetting benefits, policy overreaction (Maor 2012) is generated in the short-term and, if reinforced by positive feedback, a policy bubble is generated in the long run (Jones et al. 2014; Maor 2014a, 2020). An example is the crime and punishment bubble in the USA in the 2000s (Jones et al. 2014), especially the over-criminalization of drug use and possession, which has been manifested by over 1 million drug possession arrests by police each year (FBI 2017). However, there are conditions under which the implementation of over-designed policies may lead to proportionate policy and, in rare cases, even to policy underreaction (Maor 2014b). This is most likely to occur when the implementation of over-designed policy is (significantly) undermined by the complexity of the design. Regarding policy effectiveness, although over-designed policies are not economical and involve wasted resources, they do not necessarily preclude the achievement of policy goals. Policy over-design that is implemented as planned does not undermine performance since its results may be more robust than proportionately designed policies. However, this policy may also cost more.

By contrast, under-designed policies run the risk of failure. Once this risk materializes, it is likely to generate policy underreaction in the short-term and, if sustained by self-reinforcing processes, negative policy bubbles (Maor 2016) in the long run. Furthermore, if a policy design involving financial assets does not include strict financial regulations (hence, under-designed policy), financial bubbles are highly likely to emerge. An example is the European Union Emission Trading Scheme (EU ETS), which underestimated the fact that carbon allowances are financial assets, potentially vulnerable to the behavior of traders and investors. This scheme led to episodes of speculation and price bubbles related to energy and environmental policy announcements (Cretí and Joëts 2017). Importantly, as long as the risk of failure does not materialize, under-designed policies may be (wrongly) perceived by the public as proportionate. This is because such policies require substantial policy maintenance costs that may be hidden from public view. Under-designed policies entail also significant policy failure costs and have a short implementation due to the efforts saved. In addition to their poor operational performance, under-designed policies increase the risk of accidents, cause economic losses during negative events, and may incur liability lawsuits and voter dissatisfaction.

The fact that an under-designed policy is prone to failure while an over-designed one is prone to excessive costs may, at times, have significant political implications. This is because claims of under-design are more threatening to political executives’ reputations than claims of over-design. Allegations of policy under-design are directly targeted at political executives’ distinct contribution to the public good, whereas arguments of over-design mostly concern the negative externalities of their actions. Further, in certain policy areas, the harm of over-design is visible and salient only to concentrated groups (e.g., regulated businesses), whereas under-design has greater resonance with the public at large (e.g., Gilad et al. 2015, 457). Therefore, under-designed policies may entail a greater threat to the reputation of political executives than over-designed policies. Some elected executives are well aware of these implications, and therefore, the space of over- and under-design may be of interest to them.

How do technical problem-solving and politics interact during policy design processes?

Many policy problems have multiple and complex causes, some of which may be ideologically and politically charged. It is precisely here that politicians enter into the fray, as they consider policy instruments and consequently select their own favorites—(mostly) the same instruments all the time (e.g., choosing market approaches over administered decision-making, or anchoring agriculture subsidies on the basis of land ownership rather than environmental benefits). Here, we diverge from the assumption that the most important goal of policy design is to reduce the space of policy performance variability through reducing the space of under-design while minimizing the increase in the space of overdesign. The need to reduce the space of under-design arises primarily as a result of policy designers’ wish to minimize the risk of policy failure. The need to minimize the increase in the space of overdesign—rather than eliminate this space because it generates excessive cost—results from using over-design in order to provide margins of safe operation (Braathen 2007; Braathen and Croci 2005).

The risks of compromised reliability, following policy under-design, and excessive costs, following policy over-design, are sometimes deliberately ignored for political reasons, especially when political executives are vulnerable to voters (Maor 2017b, c, 2019a, b; Maor et al. 2017; Howlett and Kemmerling 2017). This is because deliberate over- and under-design of a technical problem by manipulating parameters relating to the quality of information, especially in politically sensitive times (elections and protests), may solve a political problem by inflicting damage on political rivals and/or successfully shaping voters’ perceptions in a favorable manner. Consequently, political executives who wish to overreact in order to signal resolve may opt for over-designed policy solutions, such as aggressive and hypervisible policy responses in the areas of counter-terrorism and crime prevention (e.g., Maor 2019c). Such policy solutions will be expensive and costly, yet they may be effective in political terms (Bovens and ‘t Hart 1996). Political executives who wish to underreact, for example, in areas of climate change and equal pay, may opt for policy solutions that are under-designed. They may do so by, for example, rejecting scientific information that counters their existing policy positions (e.g., Heikkila et al. 2020). Such under-designed solutions will not be reliable and are likely to fail, yet for some elected executives these policy failures do not entail high political costs among their core voters.

Finally, although under- and over-designed policies are suboptimal designs, they can, at times, yield significant political benefits for the elected executives who initiate them. Thus, in the vein of one person’s crisis is another’s opportunity, we facilitate here a more realistic understanding of the conditions under which a policy is under- or over-designed to respond to some objectives but entirely proportionate (or correct) with regard to others. Studying intentional over- and under-design is a new frontier in the policy design arena, bringing us to the methodological section.

Methodological considerations

Scholars may comparatively examine the (dis)proportionality of policy design by focusing on the final design for the same policy problem across countries. The extent of disproportionality can be measured by the distance between a given country’s set of proposed policy responses and the average response of the countries under investigation while accounting for domestic-level variation in the severity of the policy problem (e.g., De Francesco and Maggetti 2018). Scholars can thereafter examine the role played by information quality in how policy is under- or over-designed. They may also use the Information Quality Assessment (IQA) questionnaire, which measures information quality along 16 dimensions that were verified in different organizational contexts and by numerous studies, alongside objective measures that do not require data or policy designer’s time for the purposes of assessment (Strong 2009).

Scholars can also employ process tracing to study how elected executives, through their choice of the information provided, the information provider, or the information mediator, as well as the sequences of these choices, can influence the behavior of policy designers in pursuit of their objectives, even if they have no ability to change policy design outcomes or force the policy designers to choose particular actions that determine outcomes. Alternatively, researchers can utilize randomized survey experiments as well as laboratory experiments involving the direct manipulation of information quality. One experiment can test the hypothesis that, in cases of a complex policy problem and a complex policy solution, low-quality information influences policy designers’ interpretations of high-quality information. Another experiment can test the effect of adding more high-quality information into the policy design process. The evidence can be constructed to demonstrate that only one interpretation is correct or, alternatively that numerous interpretations are correct. The experimental groups may vary, among other differences, with regard to whether or not the identity of the information providers is visible to the respondents.

Because we do not possess a ‘logic’ or ‘model’ of explanation, we should combine elite interviews (especially using open questions)—by means of which we can comprehend the meanings, beliefs and preferences of the elected executives involved, and their thoughts about their actions—with content analysis of speeches, press releases, inquiries, hearings and other documents. Scholars should also pay attention to the context within which policy-makers operate. If researchers gain access to the policy design process, a detailed behavioral analysis should aim to describe the process via which deliberate over- and under-design are advanced, especially the ways political executives choose an information structure that facilitates disproportionate policy design.

Directions for future research

In light of the conceptualization developed here, we have identified three broad areas that offer the most promising avenues for future research concerning the role played by information quality in how policy is designed: (1) construct validity; (2) levels of analysis other than the government ministry; and (3) temporality and dynamism.

Regarding construct validity, the study of over- and under-design will benefit from research aimed at defining and dimensionalizing information quality, as well as developing empirical measures. In this respect, one dimension—the volatility of data over time—will be more important in coming years, as large data repositories containing data acquired over longer periods become increasingly important in policy design processes (e.g., Giest 2017). Improved dimensionality of information quality can bring to the fore the factors that potentially undermine information quality.

Regarding levels of analysis, policy design in central government may be a ministry-level construct, with design restricted to one ministry; an inter-ministry construct, with policy design undertaken by multiple government departments; and a multi-level construct involving policy design actors representing different levels of government. Future research should ascertain variance in the role of information quality in policy design at these levels of analysis, and it should also consider which factors and actors at each level can affect these variations, or are affected by them, in ways that increase the probability for over- and under-design. Developing multi-level theorizing and analytical techniques will be an asset for researchers and practitioners. Future research that takes advantage of these emerging approaches can enhance our understanding of how the parameters of information quality contribute to explaining under- and over-design across levels of analysis.

Regarding temporality and dynamism, rather than opting for static research concerning the role of information quality in processes of policy design, more attention should be paid to issues of temporality, both in the short and long term (e.g., Capano 2018). Temporal variability of information quality can significantly influence the effectiveness and efficiency of information-driven processes involved in policy design. Future theorizing and empirical research must pay greater attention to the role of time in such processes; how and why such processes change and evolve over time; how the changes in the role played by businesses and civil society in such processes affect policy design outcomes; and how different macro-social factors generate such processes. The high level of interest in policy design improvement among practitioners may facilitate their cooperation in advancing the study of information quality in policy design processes.