Teaching and Incentives: Substitutes or Complements?

Abstract

Interventions to promote learning are often categorized into supply- and demand-side approaches. In a randomized experiment to promote learning about COVID-19 among Mozambican adults, we study the interaction between a supply and a demand intervention, respectively: teaching via targeted feedback, and providing financial incentives to learners. In theory, teaching and learner-incentives may be substitutes (crowding out one another) or complements (enhancing one another). Experts surveyed in advance predicted a high degree of substitutability between the two treatments. In contrast, we find substantially more complementarity than experts predicted. Combining teaching and incentive treatments raises COVID-19 knowledge test scores by 0.5 standard deviations, though the standalone teaching treatment is the most cost-effective. The complementarity between teaching and incentives persists in the longer run, over nine months post-treatment.

1. Introduction

Societies devote substantial resources to helping people acquire knowledge. These efforts often take place in educational institutions. In addition, outside of school settings, there are many efforts to promote learning about financial decision-making (raising “financial literacy”), public health (promoting “health literacy”), and many other areas. Efforts to promote learning commonly take one of two approaches. First, one can teach, via classroom instruction, broadcast media, advertising, social media, or other means. Second, one can improve learners’ incentives to acquire knowledge, such as by informing them about the returns to education, or providing incentives for good performance on learning assessments (e.g., merit scholarships or other rewards based on test scores). These two broad approaches are often described as operating on the “supply” and “demand” sides of education, respectively (Banerjee and Duflo, 2011; Glewwe, 2014). Supply interventions provide educational inputs (e.g., teaching and instruction), reducing the marginal cost of learning. Demand interventions seek to raise learners’ perceived marginal benefit of learning.

Supply and demand educational interventions often operate at the same time. Existing research, however, says little about interactions between such interventions. Crucially, are supply and demand interventions substitutes or complements? Understanding complementarities between interventions is key for cost-effectiveness analyses, and thus decision-making on optimal combinations of policies (Twinam, 2017). If two interventions are complements, the gains from implementing both exceed the sum of the gains of implementing each one singly. The greater the complementarity, the more attractive it could be to implement both policies together, rather than either one alone. If they are substitutes, by contrast, the gains from implementing both are less than the sum of the gains of implementing each one singly. In this case, it becomes more likely that the optimal course would be to implement just one or the other of the policies, not both together.

We implemented a randomized controlled trial of a supply and a demand intervention to promote learning, estimating the degree to which the two are substitutes or complements. We study learning about COVID-19 among adults in Mozambique, and implement treatments that are representative examples of supply and demand interventions to promote learning. Our supply treatment teaches about COVID-19. It provides information targeted at individuals’ specific knowledge gaps, a pillar of the “teaching at the right level” (TaRL) pedagogical approach (Banerjee et al., 2007; Duflo et al., 2011). We view this feedback as an important component of teaching; however, we do not attempt to teach principles (e.g., of immunology) which would allow respondents to answer new questions correctly (“in-depth” teaching). The demand-side treatment offers individuals financial incentives for correct responses on a later COVID-19 knowledge test. This treatment is analogous to educational testing with non-zero stakes for test-takers.

Abiding by COVID-19 health protocols, we interacted with our 2,117 Mozambican study respondents solely by phone. We registered a pre-analysis plan prior to implementation. We assessed respondents’ COVID-19 knowledge in a baseline survey, and then implemented the teaching and incentive treatments in a 2x2 cross-randomized design. The design created a control group and three treatment groups: “Incentive” only, “Teaching” only, and “Incentive plus Teaching” (or “Joint”). We measure impacts on a COVID-19 knowledge test several weeks later.

To theoretically examine interactions between teaching and incentives, we write down a simple model of knowledge acquisition. Individuals can exert effort to search for knowledge on their own, and can also learn from teaching. In the model, the Incentive and Teaching treatments can be either substitutes or complements, depending on the magnitudes of two countervailing effects. The Incentive treatment has a motivation effect, potentially enhancing the impact of Teaching. But Teaching can have a crowding-out effect, by reducing the need to search for knowledge, thus lowering the effectiveness of the Incentive treatment. We define a parameter $\lambda$, representing the degree of complementarity. If motivation effects dominate crowding-out effects, then Incentive and Teaching are complements ($\lambda >0$). Otherwise, they are substitutes ($\lambda <0$).

In advance of sharing our results publicly, we determined a reasonable “benchmark” $\lambda$ by collecting expert predictions of our treatment effects. The vast majority of surveyed experts expected the two treatments to be substitutes, predicting that the effect on test scores of the combination of both treatments would be less than the sum of the effects of each treatment implemented singly. In the context of the theoretical model, expert predictors believed that when offering the Incentive and Teaching treatments together, the crowding-out effect would dominate the motivation effect.

We find substantially more complementarity than experts predicted: actual estimated $\lambda$ is positive, and highly significantly different from experts’ negative prediction of $\lambda$. The Incentive treatment raises COVID-19 knowledge test scores (fraction of questions answered correctly) by 1.56 percentage points, while Teaching does so by 2.88 percentage points. By contrast, the Joint treatment raises test scores by 5.81 percentage points, 31% larger than the sum (4.44 percentage points) of the effects of each treatment provided separately. Actual estimated $\lambda$ is also marginally statistically significantly different from zero, another benchmark of interest. These results are consistent with the theoretical case in which the motivation effect dominates the crowding-out effect when providing both treatments together. The effect of the Joint treatment is large in magnitude: 0.5 test score standard deviations. Additionally, the Joint treatment’s significant positive effect and complementarity pertain to newly asked questions (not just questions previously asked) and persist over nine months after the intervention.

We provide a simple illustration of the importance of the estimate of $\lambda$ for cost-effectiveness comparisons. We use our actual treatment effect estimates and implementation costs to calculate cost-effectiveness of the individual Incentive and Teaching treatments, as well as the cost-effectiveness of the Joint treatment for different values of $\lambda$. Our estimated $\lambda$ is below the threshold at which the Joint treatment would be the most cost-effective of our three treatments. That said, governments or NGOs implementing our treatments in different contexts may come to different cost-effectiveness rankings given their specific implementation costs.

This research contributes to economics research on education and learning. There is a substantial literature examining the impacts of supply- and demand-side educational interventions (Glewwe, 2014; Evans and Popova, 2015; Le, 2015; McEwan, 2015; Conn, 2017; Muralidharan, 2017).

On the supply side, studies have examined provision of educational supplies (Glewwe et al., 2000, 2009), school facilities (Duflo, 2001), new teaching technologies (Muralidharan et al., 2019), and “teaching at the right level” (TaRL) (Banerjee and Duflo, 2011; Duflo et al., 2011). Angrist et al. (2020) show that teaching via cellphone can offset learning loss during the COVID-19 pandemic. Mbiti et al. (2019) show complementarity between two supply-side interventions (increased school resources and teacher incentives). Outside of school settings, supply-side efforts are made to provide health education to promote “health literacy” (Batterham et al., 2016), financial education to promote “financial literacy” (see Kaiser and Menkhoff (2017) for a review), and agricultural “extension” to improve farming knowledge (Anderson and Feder, 2007; Fabregas et al., 2019).¹ Our Teaching treatment implements a targeted approach to promote COVID-19 health literacy.

Demand-side educational interventions seek to increase the perceived returns to learning. In school settings, studies have examined impacts of providing information on the wage returns to schooling (Jensen, 2010), merit scholarships based on test performance (Kremer et al., 2009; Berry et al., 2019), or incentives for test performance (Angrist and Lavy, 2009; Levitt et al., 2011; Fryer, 2011; Behrman et al., 2015; Burgess et al., 2016; Fryer, 2016; Hirshleifer, 2017). Outside of school settings, studies have evaluated incentive-based strategies such as cash payments, deposit contracts, lotteries and non-cash rewards to promote healthy behaviors (Finkelstein et al., 2019), but do not target learning outcomes. Our Incentive treatment is analogous to policies providing financial incentives for test performance, making it a rare example of a demand-side policy to promote learning among non-students.²

The most novel feature of our work is that we explicitly highlight and measure the complementarity between a supply-side and a demand-side educational intervention. Behrman et al. (2015) and List et al. (2018) study the interactions between test-score incentives for teachers (supply-side) and students (demand-side), but do not estimate a complementarity parameter, as we do.³ In addition to being of policy interest, we view this interaction as of particular theoretical interest due to the countervailing motivation and crowding-out effects of combining supply- and demand-side educational interventions.

Our study also contributes to understanding adult education in health crises. Broader research suggests that adults have higher economic and physiological barriers to learning (Aker and Sawyer, 2021), and that successful health informational interventions are comprehensive but not overly complex (Dupas et al., 2011). Additional challenges in health crises often include underlying institutional mistrust and misinformation (Vinck et al., 2019) and logistical obstacles to needs assessments and outreach with vulnerable populations (Checchi et al., 2017). In this context, we demonstrate simple interventions that can complement phone data collection during epidemics (Angrist et al., 2020; Maffioli, 2020; Magaço et al., 2021). In particular, our Teaching intervention shows that providing feedback on knowledge-based questions is a feasible and impactful add-on to health surveys—for example, on "knowledge, attitudes, and practices (KAP)" surveys common in public health.⁴

Related studies seek to improve COVID-19-related knowledge among adults. Alsan et al. (2020) show that messaging tailored to minorities improves their COVID-19-related knowledge. Mistree et al. (2021) and Maude et al. (2021) find that randomly assigned teaching interventions improve COVID-19-related knowledge in India and Thailand, respectively, while Bahety et al. (2021) find no evidence that COVID-19 SMS-based information campaigns improve knowledge in rural India. Angrist et al. (2020) and Banerjee et al. (2020) use phone-based interventions to address issues during the pandemic.

2. A Simple Model of Learning

There are N dimensions of knowledge. On each dimension there are two possible states {A, B}: a correct state A and a incorrect state B. For example, one dimension of knowledge might be “Hot tea helps to prevent Covid-19,” with the two states being “correct” and “incorrect”.

Initial Knowledge.

Every agent has independent priors on each state which we model as follows. The agent initially believes that both states are equally likely to be correct. She then receives a binary signal that informs her about the correct state – that signal is correct with probability $\mu >\frac{1}{2}$. This implies that a share $\mu$ of population have a posterior that places weight $\mu$ on the correct state while a share $1-\mu$ of the population has a posterior that places weight $\mu$ on the incorrect state.

Actions.

For each knowledge dimension $i$, an agent takes an action $x_i\in \{a,b\}:a(b)$ will provide utility 1 if the correct state is $A(B)$ and 0 otherwise. The agent will therefore always choose the action that is appropriate for the state on which she places a greater subjective probability on being correct. For example, equipped with initial knowledge a share $\mu$ of the population will derive utility 1 by taking the correct action and a share $1-\mu$ of the population will derive utility 0. The initial expected utility of agents is therefore $\mu$. Let $R$ be the benefits or returns that agents gain for knowing the correct state of a knowledge dimension.

Teaching.

Now assume that the government or some other authority seeks to teach the agent the correct state (our Teaching treatment). The agent will adopt this recommendation with probability $p(R)$ which captures the credibility of the source (and hence the agent’s propensity to follow the advice) as well as the attention she pays to the advice. Otherwise the agent ignores the recommendation.

Importantly, attention can depend on the return the agent receives for being correct: $p(R)$ is (weakly) increasing in $R$. This creates a positive interaction effect between the return to knowledge and the propensity to absorb what is taught.

Teaching generates 3 types of posteriors:

• A share $p$ of the population places subjective probability 1 on the correct state. This group is made up of all agents who followed the advice.

• A share $(1-p)\mu$ of the population places subjective probability $\mu$ on the correct state.

• A share $(1-p)(1-\mu )$ of the population places subjective probability $1-\mu$ on the correct state.

When the perceived returns to knowledge are negligible (i.e., $R=0$), the Teaching treatment increases the share of correct answers to $p(0)+(1-p(0))\mu$.

Returns to Knowledge.

Recall that agents gain benefits or returns $R$ for knowing the correct state of a knowledge dimension. She can spend effort $e\ge 0$ on searching for knowledge at a cost of $\alpha e^2$ – this will provide a correct signal with probability $e$. Then with probability $1-e$ she does not find the correct answer and follows her initial belief $\mu$. Returns $R$ may be manipulated by a learning incentive (our Incentive treatment), which increases the share of correct answers to $e^{\ast }+(1-e^{\ast })\mu$.

• Agents who already experienced the Teaching treatment and paid attention to it expend effort $e=0$ since their posterior is already placing probability 1 on the correct state. Knowledge depreciation is ignored as it is assumed to be the same, on average, for all agents.

• The other two groups of agents will in equilibrium spend the same amount $e^{\ast }$ on searching behavior. Their expected utility equals:

  $$(e^{\ast }+(1-e^{\ast })\mu )R-\alpha (e^{\ast }{)}^2$$

  The first two terms capture the utility from taking the correct action when she finds the correct signal, and the last term captures the cost of searching for correct knowledge.

  The optimal action therefore equals $e^{\ast }=\frac{R}{2\alpha }(1-\mu )$: she will search more if their initial knowledge is less precise (lower $\mu$), if searching is less expensive (lower $\alpha$) or if the reward $R$ is higher.

To summarize, the Teaching and Incentive treatments give rise to three types of posterior beliefs:

• A share $p(R)+(1-p(R))e^{\ast }$ of the population places subjective probability 1 on the correct state. This group is made up of all agents who followed the advice.

• A share $(1-p(R))(1-e^{\ast })\mu$ of the population places subjective probability $\mu$ on the correct state.

• A share $(1-p(R))(1-e^{\ast })(1-\mu )$ of the population places subjective probability $1-\mu$ on the correct state.

Learning.

The share of the population with correct knowledge prior to the Teaching and Incentive treatments is $\mu$.

After the Teaching and Incentive treatments, the share of correct answers increases to:

$$p(R)+(1-p(R))e^{\ast }+(1-p(R))(1-e^{\ast })\mu$$ (1)

We organize the share of correct answers by treatment in Table 1.

Table 1:

Test Scores and Treatment Effects Implied by Theoretical Model

Treatment	Share of Correct Answers	Boost (Versus Control)
Control	$\mu$	0
Teaching Only	$p(0)+(1-p(0))\mu$	$p(0)(1-\mu )$
Incentives Only	$e^{\ast }+(1-e^{\ast })\mu$	$e^{\ast }(1-\mu )$
Incentive plus Teaching (Joint)	$p\left(R\right)+(1-p(R\left)\right)e^{\ast }+(1-p(R\left)\right)(1-e^{\ast })\mu$	$p\left(R\right)(1-\mu )+e^{\ast }(1-\mu )-e^{\ast }p(1-\mu )$

We can now compare the effect of the Incentive plus Teaching (Joint) treatment with the simple sum of each treatment implemented separately. Let this difference be defined as the complementarity parameter $\lambda$:

$$\lambda \equiv \text{Joint}-(\text{Teaching only}+\text{Incentive only})=\underset{\text{motivation}}{\underbrace{(p(R)-p(0))(1-\mu )}}-\underset{\text{crowding out}}{\underbrace{e^{\ast }p(1-\mu )}}$$ (2)

There are two opposing effects. The motivation effect captures that Teaching has greater impact when the return to knowledge is higher (e.g., because agents are more motivated to learn, she pays more attention to teaching, or exert more knowledge-search effort). On the other hand, there is a crowding out effect because Teaching reduces the need to search for knowledge and hence the effectiveness of the Incentive treatment.

Lemma 1 The Teaching and Incentive treatments are complements if the motivation effect dominates the crowding out effect. Otherwise, the Teaching and Incentive treatments are substitutes.

When the Teaching and Incentive treatments are complements, the complementarity parameter will be positive: $\lambda >0$. When they are substitutes, on the other hand, it will be negative: $\lambda <0$. When $\lambda =0$, we say the two treatments are additive.

In our empirical analyses, we provide an estimated complementarity parameter, $\widehat{\lambda }$.

3. Sample and Data

3.1. Data

We implemented three rounds of surveys by phone in July-November 2020: a pre-baseline, baseline and endline survey (see Figure 1 for a study timeline). Respondents were from households with phones in the sample of a prior study (Yang et al., 2021).⁵ We surveyed one adult per household. Participants received a small gift of 50 meticais (approx. US$0.70) after completing each survey, as explained at study enrollment, which was transferred via MPesa over 93% of the time and phone credit recharge otherwise. Appendix A provides details on the COVID-19 context and study communities.

Figure 1: Study Timeline.

Notes: Pre-analysis plan uploaded and treatments randomly assigned immediately prior to start of baseline survey, on Aug. 25, 2020. Treatments implemented immediately following baseline survey on same phone call. There was at least a three week gap between baseline and endline survey for any given study participant. Not depicted is the post-endline survey implemented between June 30 and August 30, 2021 that we use in the long-run analysis described in Section 5.4.

Between a pre-baseline survey and baseline survey, we randomly assigned households to treatments and registered a pre-analysis plan (PAP). The baseline survey was immediately followed by over-the-phone treatment implementation. There was a minimum of 3.0 weeks and average of 6.3 weeks between baseline and endline surveys for all respondents. Baseline and endline surveys occurred when COVID-19 cases were rising rapidly.

The endline sample size is 2,117 respondents, following a sample size of 2,226 at baseline. The retention rate between baseline and endline is 95.1% overall, at least 94.4% in each of the seven districts surveyed, and balanced across treatment conditions.

We measured respondents’ COVID-19 knowledge in three categories: 1) general knowledge (risk factors, transmission, and symptoms); 2) preventive actions (preventing spread to yourself and others); and 3) government policies (official actions taken by the national government of Mozambique). Pre-baseline, we tested numerous pilot questions. Then, at baseline and endline, we administered a pre-specified set of knowledge questions and their correct responses in our analysis plan submitted to the AEA RCT Registry. At baseline, we asked respondents knowledge questions randomly selected within each category, and respondents randomly assigned to the Teaching treatment were given feedback on incorrect and correct responses. At endline, respondents were asked a full set of knowledge questions to estimate treatment effects. Poor internet access and low ownership of electronic devices make it very unlikely that respondents looked up correct answers during the questionnaire. See Appendix B for details on question selection and the list of questions.⁶

3.2. Outcomes

Outcomes are COVID-19 knowledge test scores: the share of knowledge questions answered correctly. Responses are considered “correct” if they match the pre-specified correct answer and are “incorrect” otherwise. At baseline, each respondent was assigned a randomized subset of 20 out of 40 questions, distributed as follows across categories: 6 (out of 12) general knowledge, 8 (out of 16) preventive action, and 6 (out of 12) government policy questions.

We pre-specified two primary outcomes: First, the Overall test score is the share of correct answers to all 40 knowledge questions asked at endline: 12 on general knowledge, 16 on preventive actions, and 12 on government actions. In the control group (N=847), this outcome has a mean of 0.781 and a standard deviation of 0.108. Second, the Teaching- Eligible test score is the share of correct answers to the 20 knowledge questions that were also asked at baseline—that is, those that were eligible for feedback via the Teaching intervention: 6 on general knowledge, 8 on preventive actions, and 6 on government actions. In the control group, this outcome has a mean of 0.784 and a standard deviation of 0.123.

Secondary outcomes include test scores for Teaching-Ineligible questions, the remainder 20 questions NOT asked of the respondent at baseline, and newly asked questions, those questions randomly not asked of the respondent at either pre-baseline or baseline.⁷ We also analyze test scores for knowledge categories: general knowledge, preventive actions, and government policies.

4. Empirical Approach

4.1. Treatments

To improve COVID-19 knowledge, we designed two interventions to be implemented at the end of the baseline survey following all baseline questions: 1) “Incentive” and 2) “Teaching”. Respondents were randomly assigned to one of four groups (probabilities in parentheses): Incentive alone (20%), Teaching alone (20%), both treatments (“Incentive plus Teaching” or “Joint”) (20%), or a control group (40%). Randomization was stratified within 76 communities. We describe the treatments briefly below. Complete implementation protocols can be found in Appendix C.

Incentive treatment:

We informed respondents that they would earn 5 Mozambican meticais (approx. US$0.07) for every correct response to previously-asked and newly-asked COVID-19 knowledge questions on the endline survey. They were also told that this would allow them to earn 200 meticais (approx. US$2.80), if they answered all 40 questions correctly, in addition to their 50 meticais survey completion gift. 250 meticais is equivalent to half of the sample median pre-pandemic (February 2020) weekly household income. After endline questioning, the number of correct answers and resulting payment were automatically calculated in SurveyCTO, displayed for enumerators, read to respondents, and added to the 50 meticais survey completion gift.

Teaching treatment:

We provided respondents feedback on 80% of their incorrect answers and 20% of their correct answers, on average, to COVID-19 knowledge questions from the baseline survey. Feedback consisted of reminding respondents of their answer, telling them if they were correct or incorrect, and then telling them the correct answer.⁸

Joint treatment:

We informed respondents of the Incentive treatment first, then implemented the Teaching treatment.

Sample sizes by treatment condition were as follows: Incentive (N=414, 19.6% of sample), Teaching (N=418, 19.7%), Joint (N=438, 20.7%) and control group (N=847, 40.0%). In Appendix D, we show that attrition between baseline and endline is low (4.9%) and balanced across treatment conditions. We also show that chance imbalance between the baseline outcome and the standalone Incentive treatment is heavily concentrated in only one district, and that our results are robust excluding it. Finally, we show that baseline measure of household income, food insecurity, and presence of an older adult in the household are balanced across treatment conditions.

Randomization of the Incentive, Teaching, and Joint treatments was also stratified by two cross-randomly assigned treatments to improve social distancing as part of a separate study (Allen IV et al., 2021): 1) misperceptions correction, which updated beliefs upwards or confirmed beliefs about high rates of community support for social distancing, and 2) leader endorsement, which reported to respondents previously collected social distancing endorsements by community opinion leaders. In Appendix E, we present regression results showing no meaningful interactions between the social distancing treatments and this paper’s treatments. We also verify that our primary treatment effect estimates are very similar when the Test Score outcome measure excludes social distancing knowledge questions, which are most susceptible to being affected by the social distancing treatments.

4.2. Regression

As pre-specified, we estimate the following OLS regression equation:

$$Y_{i,j,t=3}={\beta }_0+{\beta }_{1}Incentiv{e}_{ij}+{\beta }_{2}Teachin{g}_{ij}+{\beta }_{3}Join{t}_{ij}+\eta {\mathbf{B}}_{ijt}+{\gamma }_i+{\epsilon }_{ij}$$ (3)

where $Y_{i,j,t=3}$ is the COVID-19 knowledge test score for respondent $i$ in community $j$. $Incentiv{e}_{ij}$, $Teachin{g}_{ij}$, and $Join{t}_{ij}$ are indicator variables for inclusion in each treatment group. ${\mathbf{B}}_{ijt}$ is a vector representing the share of correct answers to questions asked at pre-baseline and baseline, respectively.⁹ ${\gamma }_i$ are community fixed effects, and ${\epsilon }_{ij}$is a mean-zero error term. We report robust standard errors.

Due to treatment random assignment, coefficients ${\beta }_1$, ${\beta }_2$, and ${\beta }_3$ represent causal effects of the respective treatments on test scores. We estimate the complementarity parameter as a linear combination of regression coefficients: $\widehat{\lambda }={\beta }_3-({\beta }_1+{\beta }_2)$.

4.3. Hypotheses

We hypothesize that each treatment has a positive effect on test scores. Specifically, as pre-specified, we hypothesize that the coefficient ${\beta }_1$ in a regression of the Overall test score, and the coefficients ${\beta }_2$ and ${\beta }_3$ in a regression of the Teaching-Eligible test score will be positive. We adjust p-values for multiple hypothesis testing across these three coefficients.¹⁰

Additionally, using our estimated $\widehat{\lambda }$, we test the following null hypotheses: $\lambda =-0.0256$ (the mean of expert predictions, $\tilde{\lambda }$), and $\lambda =0$.

4.4. Pre-Specification

Prior to baseline data collection, we uploaded our pre-analysis plan (PAP) to the AEA RCT Registry.¹¹ In this paper, we report on a subset of analyses pre-specified in the PAP. In Appendix E, we present the “Populated PAP” for our pre-specified primary analysis. These results are substantively duplicative of and yield very similar conclusions to the primary analyses we present here in the main text.

Hypotheses related to the complementarity parameter $\lambda$ were not pre-specified in the PAP. The motivations for testing them are the theoretical model’s ambiguous prediction as to whether $\lambda$ should be positive or negative, and the fact that the vast majority of experts predicted that $\lambda <0$.

4.5. Expert Predictions

In advance of presenting our results publicly, we surveyed subject-matter experts on their expectations of our treatment effects.¹² The expert prediction survey provided respondents with an overview of the project, specifics of each intervention, and definitions of the primary outcomes (summarizing information available in the pre-analysis plan) as well as the control group mean and standard deviation for those outcomes. The survey then asked respondents to report their prediction of each treatment effect as a percentage point difference with respect to the control group mean (positive values representing positive treatment effects, and negative values representing negative treatment effects).

Experts were asked to predict the treatment effect on test scores (fraction of questions answered correctly). For the Incentive treatment, experts were asked to predict the treatment effect on the endline test score for all 40 questions asked. For the Teaching and Joint treatments, experts were asked to to predict the treatment effect on the endline test score for the 20 knowledge questions randomly selected at baseline that were eligible the Teaching treatment.

We received expert predictions from 67 survey respondents before the survey closed on January 2, 2021. Of these, 73% of respondents were in the field of economics, 45% were faculty members (most others were graduate students), and 57% had experience working on a randomized controlled trial.

Table 2 summarizes the expert predictions. To be consistent with the figures and tables in this paper, we display the predictions as fractions (bounded by 0 and 1) rather than percentage points. On average, respondents expected that Incentive would increase the test scores by 0.040, Teaching would increase test scores by 0.046, and Joint would increase test scores by 0.059.

Table 2:

Expert Predictions

Expert Prediction	Mean	Std. Dev.	Min	Max
Incentive Treatment Effect	0.0399	0.0256	0.0000	0.1000
Teaching Treatment Effect	0.0455	0.0307	−0.0196	0.1007
Joint Treatment Effect	0.0589	0.0296	0.0000	0.1200
Complementarity parameter ($\lambda$)	−0.0265	0.0333	−0.1108	0.0426
Indicator: Incentive and Teaching treatments are substitutes ($\lambda <0$)	0.8060	0.3984	0.0000	1.0000

Notes: 67 experts provided predictions on the Social Science Prediction Platform (socialscienceprediction.org) prior to knowing results. Survey closing date January 2, 2021.

For each expert who provided predictions, we calculate the complementarity parameter implied by their predictions: Predicted Joint Effect − (Predicted Incentive Effect + Predicted Teaching effect). ¹³ We refer to the average of expert-predicted complementarity parameters as $\tilde{\lambda }$. This average is negative ($\tilde{\lambda }=-0.0256$). The vast majority of experts (80.6%) expect the interventions to be substitutes, predicting that the joint treatment effect would be less than the sum of the standalone treatment effects. There is no significant difference in predicting that the interventions are substitutes across respondents who are or are not in the field of economics, faculty members, or have worked on a randomized controlled trial.

Figure 2 displays probability density functions (PDFs) of the predictions. For each treatment, the vast majority of experts predicted positive effects. The mean Incentive treatment effect (${\beta }_1$) is 0.040, while for Teaching (${\beta }_2$) it is 0.046. Notably, the mean predicted effect for the Joint treatment (${\beta }_2$) is 0.059, lower than the sum of the mean predictions for the separate Incentive and Teaching treatments (0.086): experts expect the treatments to be substitutes rather than complements.

Figure 2: Distributions of Expert Predictions of Treatment Effects and Complementarity Parameter.

Notes: Probability density functions of predicted treatment effects of 67 experts surveyed prior to results being publicized (survey closing date Jan. 2, 2021). Experts predicted effects of “Incentive”, “Teaching”, and “Incentive plus Teaching” (“Joint”) treatments on COVID-19 knowledge test score (fraction of questions answered correctly). Expert-predicted $\lambda$ values are calculated from each expert’s predictions. Mean of expert-predicted $\lambda$ values is $\tilde{\lambda }=-0.0265$. Smoothing uses Epanechnikov kernel with bandwidth 0.9924.

Graphically, the expectation of substitutability can be seen in the fact that the PDF of the Joint treatment has considerable overlap with the PDFs of Incentive and Teaching. Relatedly, in the figure we also display the complementarity parameter implied by each expert’s predictions. For each expert, we take their predicted Joint treatment effect and subtract the sum of their predictions for the separate Incentive and Teaching treatments. The distribution of experts’ $\lambda$ estimates is the gray dotted line. Most of the mass of $\lambda$ estimates lies to the left of zero: 81% of experts predicted negative $\lambda$. The mean of experts’ $\lambda$ estimates is −0.0265. We refer to this mean as $\tilde{\lambda }$, and will test the null that our estimated $\widehat{\lambda }$ equals $\tilde{\lambda }$.

5. Results

5.1. Primary Analysis

Table 3 presents the results from testing this paper’s primary hypotheses. In Column 1, we test our first pre-specified primary hypothesis regarding the effect of the Incentive treatment on the overall test score.¹⁴ The Incentive treatment has a positive effect, and is statistically significantly different from zero (p-val=0.0003) after multiple hypothesis testing (MHT) adjustment. The point estimate indicates a 0.020 increase, relative to the 0.781 mean control group test score. This effect is substantial in magnitude, amounting to 0.19 standard deviations of the outcome variable.

Table 3:

Treatment Effects on COVID-19 Knowledge Test Scores

VARIABLES	Overall (1)	Teaching-Eligible (2)	Teaching-Ineligible (3)	Newly-asked (4)
Incentive	0.0200*** (0.0054) [0.0003]	0.0156*** (0.0060)	0.0244*** (0.0069)	0.0209*** (0.0081)
Teaching	0.0160*** (0.0055)	0.0288*** (0.0064) [0.0003]	0.0032 (0.0069)	0.0017 (0.0078)
Incentive plus Teaching	0.0496*** (0.0055)	0.0581*** (0.0060) [0.0003]	0.0410*** (0.0069)	0.0416*** (0.0080)
$\widehat{\lambda }$	0.0136 (0.0084)	0.0137 (0.0095)	0.0134 (0.0104)	0.0189 (0.0120)
Observations	2,117	2,117	2,117	2,117
R-squared	0.319	0.333	0.201	0.150
Control Mean DV	0.781	0.784	0.778	0.777
Control SD DV	0.108	0.123	0.125	0.144
p-value: $\lambda =0$	0.1048	0.1462	0.1956	0.1145
p-value: $\lambda =\text{-}0.0265$	0.0000	0.0000	0.0001	0.0002
p-value: Incentive = Teaching	0.5292	0.0713	0.0069	0.0332
p-value: Incentive = Joint	0.0000	0.0000	0.0351	0.0235
p-value: Teaching = Joint	0.0000	0.0001	0.0000	0.0000

Notes: Column 1: COVID-19 Knowledge Overall test score, the share of correct answers to 40 knowledge questions asked at endline that were also randomly selected for the respondent to answer at baseline. Column 2: Teaching-Eligible test score, the share of correct answers to 20 knowledge questions asked at baseline and hence eligible for all treatments. Column 3: Teaching-Ineligible test score, the share of correct answers to 20 knowledge questions NOT asked at baseline and hence NOT eligible for the Teaching intervention. Column 4: Newly-asked test score, the share of correct answers to the 20 or fewer endline knowledge questions that were NOT randomly asked of the respondent at either pre-baseline or baseline. $\lambda$ is the complementarity parameter (see Section 2). $‘‘\widehat{\lambda }’’$ is coefficient on “Incentive plus Teaching” (“Joint”) minus sum of coefficients on “Incentive” and “Teaching”. All regressions include community fixed effects and controls for pre-treatment (pre-baseline and baseline) test scores. Robust standard errors in parentheses. Significance levels in Columns 1 and 2 adjusted for multiple hypothesis testing across the three coefficients estimated (on Incentive, Teaching, and Joint treatments); p-values adjusted for multiple hypothesis testing in square brackets. *** p<0.01, ** p<0.05, * p<0.1.

In Column 2, we test our remaining pre-specified primary hypotheses on the effect of the Teaching treatment and Joint treatment on the Teaching-Eligible test score.¹⁵ Coefficient estimates in Column 2 indicate that the Teaching and Joint treatments each also have positive effects. The point estimate on Teaching indicates a 0.0288 increase (0.23 standard deviations of the outcome variable), while the Joint treatment causes a 0.0581 increase (0.47 standard deviations). Each of these coefficient estimates is statistically significantly different from zero (p-val=0.0003 for each) after MHT adjustment.

In Column 3, we also estimate treatment effects on the Teaching-Ineligible test score.¹⁶ The Incentive intervention, which applied to newly-asked questions, indeed maintains a significantly positive effect; however, the Teaching treatment does not, suggesting that the intervention is effective in teaching specific facts but not related information on a topic. Finally, the Joint intervention maintains a significant but smaller positive effect.

For our analysis of treatment complementarity, we choose to use results on the Teaching- Eligible test score in Column 2, which contains two of our three pre-specified treatment effects. Also, as its outcome is based on questions that were eligible for all interventions, it maximizes the comparability of treatment effects across our treatment conditions.¹⁷ The fourth row of the table displays the estimate, $\widehat{\lambda }$, of the complementarity parameter, and its standard error. In Column 2, $\widehat{\lambda }=0.0137$, indicating that the Teaching and Incentive treatments are complements, rather than substitutes. The key benchmark is the mean of the expert predictions, $\tilde{\lambda }=-0.0265$. We reject the null that $\lambda =-0.0265$ (p-val<0.0001).

We also display the p-value of the test that $\lambda =0$, which is 0.1460 in Column 2. Given the standard error on $\widehat{\lambda }$, we can reject at the 95% confidence level that $\lambda <-0.0048$ (in other words, we can reject all but a very small amount of substitutability between the two treatments).

We also present these results on the Teaching-Eligible test score in Column 2 graphically. In Figure 3, we display the estimates of the three treatment effects, Joint treatment effects implied if $\lambda$ took on the values of 0 or −0.0265, and p-values of relevant tests of pairwise differences. In Figure 4, we present cumulative distribution functions of test scores by treatment group, showing that the Joint treatment leads to the largest rightward shift of the test score distribution.

Figure 3: Treatment Effects and Test of Complementarity Parameter $\lambda$ Against Benchmark Values.

Notes: Panel (a) dependent variable on y-axis is the Teaching-Eligible test score (share of correct answers to knowledge questions asked at baseline and hence eligible for all treatments). Panel (b) dependent variable is Teaching-Ineligible test score (share of correct answers to knowledge questions NOT asked at baseline and hence NOT eligible for the Teaching intervention). Bars in first three columns display regression coefficients representing treatment effects (and 95% confidence intervals) for “Incentive”, “Teaching”, and “Incentive plus Teaching” (“Joint”) treatments. Floating solid horizontal lines in fourth and fifth columns display “Incentive plus Teaching” (“Joint”) treatment effects that would be implied by different benchmark values of complementarity parameter $\lambda$. Difference between values in 3rd and 4th columns is actual estimated complementarity parameter, $\widehat{\lambda }$; the test that this difference is equal to zero tests the null that $\lambda =0$. Difference between values in 3rd and 5th columns is difference between $\widehat{\lambda }$ and mean expert prediction, $\tilde{\lambda }=-0.0265$; the test that this difference is equal to zero tests the null that $\lambda =-0.0265$.

Figure 4: Cumulative Distribution Functions of Test Score by Treatment Group.

Notes: Panel (a) dependent variable on y-axis is the Teaching-Eligible test score (share of correct answers to knowledge questions asked at baseline and hence eligible for all treatments). Panel (b) dependent variable is Teaching-Ineligible test score (share of correct answers to knowledge questions NOT asked at baseline and hence NOT eligible for the Teaching intervention). Figure displays cumulative distribution functions (CDFs) of test scores in “Control”, “Incentive”, “Teaching”, and “Incentive plus Teaching” (“Joint”) treatment groups.

In sum, our estimates of the complementarity parameter indicate that the Incentive and Teaching treatments exhibit much more complementarity than experts predicted. We strongly reject the high degree of substitutability predicted by experts. In addition, we reject at a marginal level of statistical significance that $\lambda =0$.

This complementarity is also present when evaluating treatment effects on newly asked questions, building confidence that results are driven by actual learning and not merely rote memorization or experimenter demand effects. In Column 4 of Table 3, we run regression 3 pre-specified in our PAP as of secondary interest that replaces the outcome with the share of correct answers to endline knowledge questions that were NOT randomly asked of the respondent at either pre-baseline or baseline. Thus respondents were not previously told the answers to these questions as part of the Teaching intervention, making it less obvious what the experimenters “wanted to hear”. Both the Incentive and Joint treatments have a positive effect on the newly-asked test score (statistically significant at 1% level). Additionally, we continue to reject that $\lambda =-0.0265$ (the expert prediction) at the 1% level and $\lambda =0$ at a marginal level of statistical significance.

5.2. Cost-Effectiveness

We now illustrate how the relative cost-effectiveness of the treatments we study depends on $\lambda$. We describe the analysis briefly here, providing details in Appendix F. The key inputs are:

• Treatment effect estimates for the Incentive and Teaching treatments (${\beta }_1$ and ${\beta }_2$). The effect of the joint treatment is then ${\beta }_1+{\beta }_2+\lambda$.

• Implementation costs of each treatment, per treated beneficiary (derived from actual implementation costs in this study).

We consider cost-effectiveness of each treatment, the cost per unit (1-percentage-point) increase in the test score (lower numbers are better). For a range of values of $\lambda$ we display the cost-effectiveness of each treatment in Figure A.4. The cost-effectiveness of the Incentive and Teaching treatments are horizontal, because they do not depend on $\lambda$. The cost-effectiveness of the Joint treatment is a decreasing function of $\lambda$: the greater the complementarity of the two treatments, the more cost-effective is the Joint treatment.

The intersection of the Joint treatment line with the horizontal lines indicates the “breakeven” $\lambda \mathrm{s}$, above which the Joint treatment is more cost effective than the respective single treatment. Breakeven $\lambda$ is −0.0250 for the Incentive treatment, and 0.0290 for Teaching. The latter number is more important overall, since the Teaching treatment is the more cost-effective of the two individual treatments. $\lambda$ must be above 0.0290 for the joint treatment to be the most cost-effective of the three treatment combinations.

For reference, we also show the mean expert prediction, $\tilde{\lambda }=-0.0265$, and our empirical estimate, $\widehat{\lambda }=0.0137$. At $\widehat{\lambda }$, Joint is more cost-effective than Incentive, but not as cost-effective as Teaching. Actual costs in a scaled-up program may be different from those of our study, and could yield different cost-effectiveness rankings across treatments. In Appendix F we provide an example of alternative relative implementation costs that would lead Joint to be the most cost-effective at $\widehat{\lambda }$.

5.3. Knowledge Categories

We also estimate impacts of the treatments on Teaching-Eligible and Teaching-Ineligible test scores across the knowledge categories: general knowledge, preventive actions, and government policies. Results in Table 4 are broadly similar to the estimates in Table 3 Columns 2 and 3, though treatment effects for the Incentive and Teaching interventions are heterogeneous along different dimensions.

Table 4:

Regression of Test Score (TS) Categories on Treatments

	Teaching-Eligible Test Scores			Teaching-Ineligible Test Scores
VARIABLES	General (1)	Preventive (2)	Government (3)	General (4)	Preventive (5)	Government (6)
Incentive	0.0018 (0.0099)	0.0118 (0.0088)	0.0419*** (0.0099)	0.0174 (0.0112)	0.0249*** (0.0090)	0.0422*** (0.0110)
Teaching	0.0265*** (0.0102)	0.0234** (0.0092)	0.0299*** (0.0109)	0.0044 (0.0111)	0.0016 (0.0095)	0.0146 (0.0111)
Incentive plus Teaching	0.0415*** (0.0103)	0.0535*** (0.0087)	0.0749*** (0.0100)	0.0336*** (0.0106)	0.0439*** (0.0094)	0.0538*** (0.0108)
$\widehat{\lambda }$	0.0133 (0.0157)	0.0183 (0.0136)	0.0031 (0.0154)	0.0118 (0.0166)	0.0173 (0.0141)	-0.0030 (0.0165)
Observations	2,117	2,117	2,117	2,117	2,117	2,117
R-squared	0.206	0.257	0.189	0.117	0.080	0.139
Control Mean DV	0.797	0.827	0.789	0.782	0.710	0.790
Control SD DV	0.189	0.170	0.202	0.191	0.157	0.202
p-value: Incentive =Teaching	0.0354	0.276	0.309	0.313	0.0289	0.0268
p-value: Incentive =Joint	0.000845	3.64e-05	0.00254	0.193	0.0732	0.344
p-value: Teaching =Joint	0.213	0.00365	0.000135	0.0182	0.000110	0.00143

Notes: Columns 1-3: the Teaching-Eligible test scores for knowledge categories, the share of correct answers at endline to the 6 questions on general knowledge, 8 questions on preventive actions, and 6 questions on government policy, respectively. Columns 4-6: the Teaching-Eligible test scores for knowledge categories, the share of correct answers at endline to the 6 questions on general knowledge, 8 questions on preventive actions, and 6 questions on government policy, respectively. $\lambda$ is the complementarity parameter (see Section 2 of main text). $\widehat{\lambda }$ is coefficient on “Incentive plus Teaching” (Joint) minus sum of coefficients on “Incentive” and “Teaching”. All regressions also include community fixed effects and controls for pre-treatment (Rounds 1 and 2) Test Scores. Robust standard errors in parentheses. *** p<0.01, ** p<0.05, * p<0.1.

Results for the Incentive treatment vary across knowledge category. The results suggest that the Incentive treatment was least effective at increasing general knowledge (e.g., risk factors, transmission and symptoms) and most effective at increasing knowledge on government policy. As the government’s COVID-19 policies changed just prior to and during the baseline and endline surveys, one possible interpretation is that the Incentive intervention was most effective at promoting learning of relatively new or updating information.

Results for the Teaching and Joint treatment vary less across knowledge category and more so between Teaching-Eligible and Teaching-Ineligible test scores. The Teaching treatment has a significantly positive effect on all knowledge categories for Teaching-Eligible questions, but insignificant effects otherwise. The Joint treatment remains significantly positive across all regressions. The estimated complementarity parameter $\widehat{\lambda }$ appears largest (most positive) for the preventive actions subcategory (Columns 2 and 5).

5.4. Long-Run Analysis

We further estimate the longer-run effects of the treatments over nine months later, using COVID-19 knowledge questions included in a post-endline survey that had other primary aims. This analysis was not pre-specified, so results should be considered exploratory. We briefly summarize here, providing details in Appendix G.

In a post-endline phone survey from July-August 2021, we asked 1,886 respondents (89.1% retention from endline, balanced across treatment conditions) 20 pre-specified questions on general knowledge and preventive actions. We excluded government policy questions because many pre-specified questions/answers were no longer true or applicable. Respondents received the standard 50 meticais survey completion gift but were offered no other incentives. We compare endline and post-endline treatment effects on two modified Test Scores of questions assessing general knowledge and preventive actions: 1) Test Score for all relevant questions asked in each round, and 2) Test Score for the same set of relevant questions across baseline, endline, and post-endline. For robustness, we analyze both outcomes, noting that each deviate from our pre-specified primary outcome due to the exclusion of government policy questions, and only draw conclusions supported by all regression specifications.

Results are in Table A.13. The Joint treatment has positive effects on long-run COVID-19 knowledge (Columns 2 and 4, statistically significant at 1% level) in both post-endline regressions. In addition, the complementarity parameter remains positive over this longer run. We continue to reject that $\lambda =-0.0265$ (the expert prediction) at the 1% level, and in addition also reject that $\lambda =0$ (at the 5% level or better) in all specifications. These results indicate that the Joint intervention’s impact, and the complementarity between Incentives and Teaching, were not merely short-run phenomena.

6. Conclusion

When governments and educational institutions seek to promote knowledge acquisition, two approaches are common. First, they can teach the knowledge in question (a “supply” educational intervention). Second, they can provide incentives for learners to acquire the knowledge (an educational intervention on the “demand” side). This paper is among the first to examine the interaction between a supply-side and a demand-side intervention to promote knowledge gains, estimating a complementarity parameter ($\lambda$).

We implemented a randomized study among Mozambican adults studying whether a teaching and an incentive treatment are substitutes or complements in promoting learning about COVID-19. Most experts surveyed in advance expected the two treatments to be substitutes ($\lambda <0$). In reality, the two treatments exhibit much more complementarity than experts predicted: we estimate $\lambda$ to be positive and statistically significantly larger than the expert prediction.

Our findings provide a key input for policy-making. We use our empirical estimates combined with actual implementation costs to rank potential treatment combinations for different values of the complementarity parameter ($\lambda$) in terms of their cost-effectiveness (cost per unit gain in knowledge). We identify a threshold value of $\lambda$, above which it makes sense to implement both the Incentive and Teaching treatments, rather than just one or the other. Our actual estimate of $\lambda$ does not exceed this threshold, implying that the Joint treatment is not the most cost-effective policy; rather, the Teaching treatment is. This conclusion about relative cost-effectiveness may vary in other contexts with different implementation costs.

Future studies should gauge the generality of these findings. For example, they should measure the complementarity between teaching and incentive treatments in stimulating learning about other topic areas (for example, personal finance, legal rights, or agricultural techniques); motivating behavior change;¹⁸ and in other study populations (e.g., students). It would also be valuable to examine the complementarity between other types of “demand” and “supply” interventions, particularly demand interventions that are more readily scalable than monetary payments,¹⁹ or supply interventions that involve more actors (e.g., teachers) than our standardized enumerator-led phone-based interventions. We view these as promising directions for future research.