Computer-assisted learning in the real world: How Khan Academy influences student math learning

Significance

Research has highlighted the promise of computer-assisted learning (CAL), potentially offering both learning gains and affordable scalability. However, such evidence is often based on idealistic settings, including high usage targets and intensive supports. Instead, we use large administrative data to study Khan Academy’s CAL platform in a real-world setting, where schools and teachers have more flexibility in usage. Exploiting within-teacher variation in CAL use over three years, we estimate incremental effects of CAL usage on student performance. We find even marginal CAL use improves student math performance and that students with higher prior achievement benefit more. This research supports the potential of CAL as an effective tool to support student learning in everyday classrooms—promising given its relatively low cost.

Abstract

Computer-assisted learning (CAL) offers an affordable way to implement a mastery learning approach in the classroom. However, while experimental research suggests CAL can enhance student outcomes, such findings often rely on experimental conditions not easily replicated in ordinary classroom settings (e.g., opt-in participation, extensive training and support, and high CAL usage targets). To assess the real-world impact of CAL, we draw on a large three-year panel of administrative data covering over 200,000 students in school districts that licensed Khan Academy’s Measures of Academic Progress accelerator, a program designed to support math learning. To identify causal effects, we exploit within-teacher and within-school changes in average classroom CAL practice time—a strategy that yields precise, policy-relevant estimates even at modest usage levels. We find that a classroom with 6.6 h of annual Khan Academy practice (about 11 min per week) experiences a $+$0.031 SD gain in math test score performance compared to no practice. For classrooms with higher usage levels, we find approximately linear gains, with projected effects rising to $+$0.085 SD at the recommended 30 min per week. Higher-achieving students benefit most, in part because they spend more time on CAL and progress through more skills than lower-performing peers. Teachers might reduce achievement gaps and boost overall gains by encouraging more productive use of the platform (focused on skill mastery)—especially among struggling students.

There is substantial evidence that personalized, adaptive tutoring by human instructors leads to sizable gains in student learning (1, 2). However, implementing such individualized support is challenging in standard classroom settings, where a single teacher must address the diverse skill levels and learning paces of 20 or more students simultaneously. One solution is to support classroom learning with individual or small group human tutoring. While there has been recent interest in such supplemental programs as a tool for addressing COVID-19 learning loss (3, 4), such programs tend to be costly, ranging from hundreds to thousands of dollars per student, limiting their feasibility as long-term solutions in public education.

Digital interventions like computer-assisted learning (CAL) and intelligent tutoring systems (ITS) offer an affordable, scalable way to implement personalized, adaptive learning. Using these tools, students individually practice skills at their level and receive immediate feedback. This educational technology can help instructors manage classes with students at a wide range of skill levels more effectively. Recent advances in AI (large language models and generative AI) are expected to further increase the capabilities of these tools.

Randomized control trials (RCTs) have demonstrated that some CAL and ITS tools can significantly improve student learning—particularly in math (5–14). However, such experiments are conducted in specific settings and contexts, which may limit their external validity. Many RCTs rely on short-term interventions with small samples (e.g., refs. 15 and 16), measure treatment effects based on high CAL usage targets (e.g., refs. 17 and 18), or embed CAL within broader interventions, such as additional teacher training or curriculum enhancements (e.g., refs. 8, 11, and 13). Further, most RCTs either estimate an intent to treat effect or a treatment-on-the-treated effect based on a researcher-defined minimal usage threshold. In real-world settings, teachers have greater discretion in deciding how and when to implement computer adaptive learning (19), and evidence suggests that the vast majority students who use CAL do so far less than platform recommendations and usage seen in prior research (20). Questions thus remain: At realistic (lower) usage levels, and when implemented without broader interventions, do students still benefit from CAL? How does variable usage influence learning outcomes?

Answers to these questions remain unclear. One meta-analysis found that CAL effect sizes tend to shrink in larger-scale studies (21), possibly reflecting lower implementation fidelity at scale. Further, most experimental work does not consider, nor is powered to detect, the impact of usage on student outcomes.^* This is important as low or inconsistent usage may significantly dampen, or even negate benefits identified via RCT (CAL typically involves activities such as logging and navigating new software interfaces, which may take away from other classroom activities). For example, in a large-scale reading technology study, students averaged just one-third of the recommended intervention hours, resulting in no significant gains (23). In addition, research suggests that teacher supports can substantially enhance CAL effectiveness (18); however, supports are often limited in everyday classrooms.

In this paper, we leverage large-scale administrative data from Khan Academy (KA) to identify the average impact of CAL platform use on student math learning in real-world settings. Understanding this relationship can help education stakeholders understand both the potential benefits of CAL and the conditions needed to achieve results. However, such estimation is challenging. While previous observational research has provided valuable insights, the unbiased causal effect of varied CAL usage has not yet been compellingly established.

In observational settings, there is often substantial variability in individual student CAL usage, but this variability is likely related to unobserved latent factors, confounding causal estimation. Prior work has measured correlations between, or built predictive models linking, CAL usage and student outcomes (e.g., refs. 12 and 24–26), but such papers do not identify causal effects. Recent work has used difference-in-difference (27 via reading scores) and matching (28) approaches to analyze observational data from a single year to estimate treatment effects. While promising, these analyses still suffer from confounding variables (like students that prefer math to reading or who differ in motivation). Most recently, researchers used panel data to include student and teacher fixed effects (29) but did not account for time-varying student factors. We find (SI Appendix, Tables A5 and A6) that estimating the causal impact of CAL using a student’s own usage, even when including student fixed effects, yields biased results due to unobserved, time-varying confounders.

This study advances the literature by using a more rigorous estimation strategy to identify the causal impact of CAL usage on student math learning. We assemble a three-year panel dataset including 490,000 observations, 214,000 students, 11,000 teachers, and 1,000 schools across the United States. Leveraging within-teacher variation in CAL usage over time, we estimate the incremental average treatment effect of each additional CAL hour per school year. Rather than estimate how a student’s own CAL use affects their math learning, a relationship prone to hidden confounders, we estimate the impact of CAL usage based on variation in teacher use over time. Specifically, we exploit changes in CAL usage among other students taught by the same teacher or within the same school. Our multiyear panel data allow us to control for student, teacher, and school fixed effects when identifying estimated treatment effects. In addition to our central analysis, we extend our approach to investigate heterogeneous treatment effects.

While we focus on one CAL platform (Khan Academy), our approach is likely generalizable to other large-scale intelligent tutoring system and computer assisted learning programs. At its core, KA identifies each student’s current skill level, provides immediate feedback as the student works through problems, and promotes mastery learning by encouraging students to demonstrate proficiency in foundational concepts before progressing to more advanced skills. Other platforms that effectively implement these core competencies are likely to produce similar learning gains.

Data

Measures of Academic Progress (MAP) are assessments developed by the Northwest Evaluation Association (NWEA). MAPs are vertically scaled, allowing them to capture learning gains for students who are far below or above grade level. MAP Accelerator is a digital learning tool for students in grades three through eight, developed by KA and NWEA. It uses students’ MAP scores to design an individualized KA curriculum. We use data from KA on students in districts that license the MAP Accelerator.^† Students in the sample are generally lower performing than the national average. Students who do not spend any time using KA remain in the administrative data, as long as they took the baseline and endline math tests.

When a school district licenses MAP Accelerator, all students in the district are mandated to take a start-of-year (baseline) and end-of-year (endline) MAP assessment in both math and reading. Depending on the teacher and school, students may do activities that are suggested from their MAP Accelerator and/or in the general KA library. The data include test scores, any (MAP accelerator or general) KA platform usage, and student- and school-level characteristics (e.g., sex, ethnicity; proportion of students eligible for free and reduced lunch).

Our empirical strategy relies on variation in within-teacher use of KA over time; as such, we generate a panel dataset by restricting a repeated cross-sectional dataset across three school years (2020/21 to 2022/23) to include teachers who we observe in at least two school years and all observations of students who have one of these multiyear teachers in at least one year. We then remove “singletons:” students, teachers, and schools with only one observation in the panel dataset. See SI Appendix, section A1 for additional details. Usage varies significantly but is low relative to existing research and platform recommendations: In our final panel, 24% of students have zero platform usage, 65% spend under 10 min per week (6 h per year), and only 10% meet KA’s recommendations of 18 h a year (approximately 30 min per week).

Methods

Estimating Average Treatment Effects.

Our goal is to estimate how a student’s annual CAL use affects their math learning. We measure CAL platform usage in two ways: 1) learning math hours and 2) math skills progressed to proficient on the platform. The outcome variable is standardized math learning: the change in MAP test scores from baseline (start-of-school-year) to endline (end-of-school-year) tests, standardized within grade and school year. The entire repeated cross-section is used when standardizing the difference in MAP test scores. Directly predicting math outcomes using student-level CAL usage will yield biased estimates: Time spent on the platform is endogenous, likely influenced by latent unobserved factors such as their math interest and aptitude.

To address student endogeneity, our preferred specification proxies each student’s annual CAL usage with average usage among all other students taught by the same math teacher that year, leaving out the student in question. Illustrated in Fig. 1, while there is variation, students’ own CAL usage is concentrated around their teacher’s leave-student-out average in both absolute levels (SI Appendix, Fig. A3) and year-to-year changes.

Fig. 1.

Distribution of change in student hours given change in teacher mean. Note: In this figure, continuous x-axis values are binned into five hour buckets. For each bin, the y-axis distribution is reported as a violin plot.

However, while leveraging leave-student-out average CAL usage among students with the same teacher addresses student-level endogeneity, teacher-level endogeneity remains: More effective teachers might encourage more overall CAL usage. To address this and other potential static effects, we exploit our panel data and include a range of fixed effects in our ordinary least squares regression:

$$\begin{array}{c}\hfill SM{L}_{\mathit{sktgy}}=\beta {\overline{\mathit{LMH}}}_{ty,-s}+{\alpha }_s+{\lambda }_k+{\gamma }_t+{\varphi }_{\mathit{gy}}+{\epsilon }_{\mathit{sktgy}},\end{array}$$ [1]

where $SM{L}_{\mathit{sktgy}}$ is standardized math learning for student $s$ in school $k$ under math teacher $t$ in grade $g$ and year $y$, ${\overline{\mathit{LMH}}}_{ty,-s}$ is average learning math hours among all students under math teacher $t$ in year $y$, excluding student $s$; and (${\alpha }_s,{\lambda }_k,{\gamma }_t,{\varphi }_{\mathit{gy}}$) are student, school, teacher, and year-grade fixed effects. To account for potential serial correlation among students taught by the same math teacher or attending the same school, we apply two-way cluster-robust SEs, clustering at both the teacher and school levels. This allows us to control for time-invariant student, school, and teacher quality. As such, to identify treatment effects, we rely on variation in within-teacher usage of computer assisted learning over time. To estimate the impact of skills mastered, we replace ${\overline{\mathit{LMH}}}_{ty,-s}$ with ${\overline{\mathit{MSP}}}_{ty,-s}$: Average math skills progressed to proficient on the platform among all students under teacher $t$ in year $y$, excluding student $s$.

Interpreting $\beta$ as the causal effect of CAL usage requires an identifying assumption: Changes in within-teacher use of CAL over time are independent of other time-varying factors that affect student learning. This is plausible: Many exogenous factors influence teachers’ usage decisions, such as shifts in school policy, receiving new professional development on how to use CAL, and a teacher’s own beliefs about the effectiveness of a particular CAL platform. Teachers may also use CAL to free up time for grading or administrative tasks in years when they are more busy. SI Appendix, Fig, A4 shows that teachers in the same schools exhibit similar usage patterns, suggesting changes are plausibly driven by external factors rather than by a teacher’s evolving quality.

However, this assumption is not guaranteed: If time-varying aspects of teacher quality simultaneously increase student learning and CAL platform use, our approach would overestimate treatment effects.^‡ Similarly, peer effects could confound the estimate: Higher average CAL usage by classmates might be driven, in part, by peers’ higher interest or ability again inflating estimated effects (30). To help validate the assumption that within-teacher CAL usage is independent of other changes that broadly influence student learning, we conduct a robustness check using standardized reading learning:

$$\begin{array}{c}\hfill SR{L}_{\mathit{sktgy}}=\beta {\overline{\mathit{LMH}}}_{ty,-s}+{\alpha }_s+{\lambda }_k+{\gamma }_t+{\varphi }_{\mathit{gy}}+{\epsilon }_{\mathit{sktgy}},\end{array}$$ [2]

where $SR{L}_{\mathit{sktgy}}$ is standardized reading learning, generated using the same method as standardized math learning (now using baseline and endline MAP reading tests). If within-teacher changes in platform use are driven by factors associated with broader time-varying quality, we would expect to see gains in both math and reading. Conversely, because the KA platform exclusively targets math learning, observing no significant impact on reading learning would support our assumption that within-teacher variation CAL usage is independent of factors influencing overall student learning.^§

While this robustness check provides some assurance of the validity of our identifying assumption, it is not sufficient to guard against time-varying, latent factors that target only math learning. We address this through a range of additional specifications and analyses. Although no single approach is immune to all potential bias, consistent results across all methods provide evidence of unbiased causality.

First, we run versions of Eq. 1 replacing teacher-level averages ($\overline{\mathit{LMH}}ty,-s$) with school-level averages ($\overline{\mathit{LMH}}ky,-s$). Under this specification, time-varying teacher factors do not bias our estimates; school-level changes influence student learning equally across teachers, independent of individual teacher quality or effort. Additionally, peer effects are much less of a concern since, unlike classroom composition, the overall student composition at the school level likely remains stable year-to-year.

However, while this specification addresses teacher-level bias and peer effects, it highlights another concern: time-varying school factors. In a particular year within a school, there may be leadership changes, budget shocks, curricula modifications, or the implementation of other math-focused initiatives. If such interventions coincide with changes in CAL usage, our estimates would reflect the combined impact of CAL and these other school-wide factors. Indeed, these factors could similarly bias our teacher-level analysis if individual teachers adjust their CAL usage in response. To address this, we replace school fixed effects (${\lambda }_k$) in Eq. 1 with school-year fixed effects (${\delta }_{\mathit{ky}}$). Under this specification, $\beta$ is identified by within-school differences in teachers’ CAL usage and within-teacher variation over time (net of school-wide changes). Consequently, school-wide yearly shocks do not bias these estimates.

We further validate our results by employing an instrumental variables (IV) approach, replacing the teacher-level proxy measure described above. Results from all IV specifications and robustness checks described above are presented in SI Appendix, sections A2–A4; they are consistent with those presented here.

To better understand the context behind year-to-year changes in a teacher’s average student usage of KA, and the counterfactual of how students’ math instruction might change when using more or less of the platform, we conducted a small survey (363 teachers in a single district, 29 responses) and a set of teacher interviews (108 contacted, 6 responded and interviewed). We discuss details in SI Appendix, sections A8 and A9.

Due to the small scale and likely nonrepresentative sample of respondents, we emphasize that survey and interview results should be largely viewed as anecdotal. Still, teacher participants reported that changes in their students’ usage across time were shaped by the teacher’s own perceptions of the platform’s effectiveness, the views of their teacher colleagues or principal, changes to school schedules that disrupted their prior Khan routine, or administration expectations or decisions (e.g. prioritizing or investing in a different CAL platform). Among teachers who decreased Khan usage, some substituted Khan class time for other CAL programs (2 survey responses, 1 interviewee) or other nonscreen forms of math practice, like collaborative activities or pen-and-paper work (4 survey responses, 4 interviewee). Teachers who increased Khan usage in the classroom reported substituting time that used to be spent on an alternate CAL platform for Khan (12 survey responses) or standard curriculum (1 interviewee). Overall usage choices seemed to be largely decoupled from other changes that might significantly influence math usage, such as increased homework. One exception is that several interviewed teachers reported year-specific school-wide changes (scheduling changes or a broad focus on math for a year); however, such school-wide changes that impact math outcomes can be accounted for by the school-year fixed effect model we consider.

Estimating Heterogeneous Treatment Effects.

We analyze whether CAL treatment effects differ by student, teacher, or school characteristics by extending Eq. 1 to include interaction terms:

$$\begin{array}{cc}\hfill SM{L}_{\mathit{sktgy}}& =\beta {\overline{\mathit{LMH}}}_{ty,-s}+{({\overline{\mathit{LMH}}}_{ty,-s}·X_{\mathit{stk}})}^{\prime }\eta \hfill \\ \hfill & +X_{\mathit{stk}}^{\prime }\delta +{\alpha }_s+{\gamma }_k+{\lambda }_t+{\varphi }_{\mathit{yg}}+{\epsilon }_{\mathit{sktgy}},\hfill \end{array}$$ [3]

where $X_{\mathit{stk}}$ is a set of indicators for student, teacher, and school characteristics. Here, we focus on two primary heterogeneous treatment effects (HTE): the impact of baseline math performance (a proxy for overall student ability) and the impact of being with a teacher whose students have a high rate of mastery progression (a proxy for effective platform usage).

Baseline math performance is measured by a student’s first-recorded math MAP score and expressed in quartiles relative to NWEA grade-level standards. If heterogeneous treatment effects of CAL usage exist across baseline math performance groups, at least two potential mechanisms may explain this variation. First, higher dosage: For each additional mean teacher math hour among other students, students in certain groups may respond by increasing their own hours of math learning and/or number of math skills progressed to proficient, beyond what is typical for other groups. A second potential mechanism is higher effects per unit of usage: Conditional on the same number of hours and skills, some students may exhibit greater benefit from such usage in terms of their standardized math learning.

To explore the extent to which differences in dosage contribute to observed heterogeneous treatment effects, we model student CAL usage as a function of their teacher’s leave-one-out average usage. We begin by estimating this relationship using the full sample, pooling all students:

$$\begin{array}{c}\hfill LM{H}_{\mathit{sktgy}}=\beta {\overline{\mathit{LMH}}}_{ty,-s}+{\alpha }_s+{\gamma }_k+{\lambda }_t+{\varphi }_{\mathit{yg}}+{\epsilon }_{\mathit{sktgy}},\end{array}$$ [4.1]

where $LM{H}_{\mathit{sktgy}}$ is the CAL math usage (in hours) of student $s$ in school $k$ with teacher $t$ in grade $g$ and year $y$. Next, we allow the relationship between teacher average usage and student CAL usage to vary by group characteristics (thereby assessing heterogeneity in dosage response):

$$\begin{array}{cc}\hfill LM{H}_{\mathit{sktgy}}& =\beta {\overline{\mathit{LMH}}}_{ty,-s}+\eta ({\overline{\mathit{LMH}}}_{ty,-s}·X_{\mathit{stk}})\hfill \\ \hfill & +\delta X_{\mathit{stk}}+{\alpha }_s+{\gamma }_k+{\lambda }_t+{\varphi }_{\mathit{yg}}+{\epsilon }_{\mathit{sktgy}}\hfill \end{array}$$ [4.2]

To consider dosage in terms of skills rather than time, we re-estimate Eqs. 4 and 5 above, replacing $LM{H}_{\mathit{sktgy}}$ with $MS{P}_{\mathit{sktgy}}$. Comparing group-specific and pooled dosage estimates allows us to assess the extent to which a particular group’s CAL usage differs from the average in response to leave-student-out teacher usage. If the only difference between students in different groups is dosage (i.e., the per-dose effect is constant across groups), then the difference in heterogeneous treatment effects from Eq. 3 and average treatment effects from Eq. 1 should be of the same relative magnitude.

Another potential source of heterogeneity is whether students—under the guidance of their teachers—are efficiently using the CAL platform. Prior work on another CAL platform has observed a strong correlation between skills advanced per hour and student learning outcomes (25). This may be influenced by how teachers implement CAL in the classroom: While some may strongly encourage their students to make progress on new math skills, others may allow time to be spent on less efficient activities. SI Appendix, Fig. A5 shows that teachers vary widely in how many new math skills their students master per hour spent on the platform. Each student’s math teacher is labeled “low” or “high” efficiency based on whether their teacher’s leave-student-out average skills per hour is below or above the median (2.60).^¶

In SI Appendix, section A5.1, we explore additional dimensions of heterogeneity, including student sex, ethnicity, and grade; prevalence of free and reduced lunch within their school; and class size. We do not find statistically significant differences in treatment effects along any of these dimensions.

Our study was approved by the Stanford Institutional Review Board and the University of Toronto Research Ethics Board.

Results

Average Treatment Effect.

Table 1 presents the estimated impact of computer assisted learning usage on external test scores (using Eqs. 1 and 2). After controlling for year-grade, school, student, and teacher fixed effects, and using leave-student-out teacher mean math hours as a plausibly exogenous proxy for individual student hours, we estimate that each additional hour of annual CAL practice improves standardized math learning by $+$0.0047 SD and each additional skill progressed to proficient improves learning by $+$0.0031 SD. Both specifications pass the robustness check that CAL usage raises math learning but does not affect reading outcomes, supporting our assumption that within-teacher variation in platform usage is independent of other factors that broadly affect student learning. Since our data sample includes limited use of CAL (SI Appendix, Fig. A1) and relatively marginal changes in CAL use over time (SI Appendix, Table A2), these estimates demonstrate that even marginal classroom CAL usage—and marginal usage changes—meaningfully influence student learning. This is encouraging: Even when CAL is implemented imperfectly, usage improves learning (and more usage improves learning more).

Table 1.

Average treatment effects

	Math learning		Reading learning
	(1)	(2)	(3)	(4)
Panel A: Coefficient estimates (per hour, per school year)
Teacher mean hours	0.0047 ***		$-$0.0007
	(0.0012)		(0.0012)
Teacher mean skills		0.0031 ***		0.0002
		(0.0005)		(0.0005)
Panel B: Model statistics
Observations	489,594	489,594	454,619	454,619
Adjusted R2	15.0%	15.0%	5.7%	5.7%

Note: This table reports the estimated impact of a math teacher increasing CAL usage among other students by one math hour or skill over a year (Eq. 1 for math learning; Eq. 2 for reading learning). Teacher means are leave-student-out. Math and reading learning are standardized within year and grade. All regressions include student, teacher, school, and year-grade fixed effects. SEs, two-way clustered at the math teacher and school level, are in parentheses. Stars indicate statistical significance: *** 0.1% level, ** 1% level, and * 5% level.

Fig. 2 uses the estimated effect of CAL usage per hour to predict the impact of adopting CAL at varying levels within the empirical range of our data. On average, students in our sample spend approximately 6.6 h per year (11 min per week) using KA, resulting in math learning gains of $+$0.031 SD annually. However, our teacher-level data allow us to reliably predict outcomes up to 15.7 h per year (26 min per week), where predicted math gains rise to $+$0.074 SD per year.^# Extending predictions beyond teacher-level support to include the full range of variation observed at the student level allows us to compare our estimates directly with prior experimental research.^‖ At 21 h per year (35 min per week), we predict math learning gains of $+$0.099 SD; 14 h (24 min per week), $+$0.066 SD. These estimates are somewhat lower than those reported by experimental studies: (18) estimate Intent-to-Treat effects of $+$0.12 to $+$0.17 SD from KA adoption, driven by classrooms averaging roughly 35 min per week of platform use; (8) found $+$0.20 SD effect on high school students using Cognitive Tutor (where the median usage was 22 h of CAL practice over a school year 34); and ref. 13 find a $+$0.18 SD effect from 14 h of CAL practice over the school year. These effects are perhaps larger than our estimates due to greater training and support accompanied with experimental interventions. On the other hand, our results align more closely when we substitute teacher-average usage with school-average usage (SI Appendix, Table A7). Using our school-level per-hour estimate of $+$0.0069 SD, predicted gains reach $+$0.14 SD at 21 h and $+$0.096 SD at 14 h per year.^**

Fig. 2.

Predicted gain in standardized math learning. Note: This figure shows predicted effects of annual CAL usage on standardized math learning. Teacher level predictions are based on regression [1]; school level predictions replace ${\overline{\mathit{LMH}}}_{ty,-s}$ with ${\overline{\mathit{LMH}}}_{ky,-s}$ (leave-student-out school average hours). Lines represent predicted gains based on coefficient estimates, plotted over the empirical support range identified using residualized variables. (Variables are residualized by estimating their relationship with student, teacher, school, and year-grade fixed effects, and extracting residuals. These residuals represent variation not explained by the fixed effects.) Solid and dashed lines represent predictions within $\pm$2.5 and $\pm$3.5 SDs of each level’s residualized distribution; dotted lines extend teacher and school estimates to the student-level support. To illustrate effects starting from zero usage, residualized support is shifted to begin at zero (e.g., $-$15 to $+$15 h would be shifted to 0 to 30 h). Note that this support is more conservative than that identified using nonresidualized data, and better reflects the variation underlying our estimates.

One concern is that our results may be driven by outliers rather than capturing the effect of marginal changes in CAL usage (large variations in teacher usage might disproportionately influence our findings). A related concern is whether impact varies nonlinearly with usage, perhaps reflecting some minimal threshold needed to see gains, or a plateauing of effects at high usage levels as found by ref. 22. To address this, we implement two residualization-based procedures. First, we separately regress standardized math learning and leave-student-out teacher CAL hours on student, teacher, school, and year-grade fixed effects, obtaining residuals (${\widetilde{\mathit{SML}}}_{\mathit{sktgy}}$ and ${\widetilde{\mathit{LMH}}}_{ty,-s}$ respectively)—these residuals represent variation not explained by fixed effects (i.e., the variation we are using to identify treatment effects). Second, using these residualized variables, we estimate two alternative regressions: one with CAL hours grouped into $B$ bins (imposing no parametric assumptions) and another including quadratic and cubic terms (imposing less restrictive assumptions compared to linear estimation):

$$\begin{array}{c}\hfill {\widetilde{\mathit{SML}}}_{\mathit{sktgy}}={\displaystyle \sum _{b=1}^B}{\beta }_b\mathbf{1}\{{\widetilde{\mathit{LMH}}}_{ty,-s}\in \text{Bin} b\}+{\epsilon }_{\mathit{sktgy}}\end{array}$$ [5.1]

$$\begin{array}{c}\hfill {\widetilde{\mathit{SML}}}_{\mathit{sktgy}}={\widetilde{\mathit{LMH}}}_{ty,-s}+{\widetilde{\mathit{LMH}}}_{ty,-s}^2+{\widetilde{\mathit{LMH}}}_{ty,-s}^3+{\epsilon }_{\mathit{sktgy}}\end{array}$$ [5.2]

Binned regression results shown in Fig. 3 confirm that even small within-teacher fluctuations in classroom CAL usage impact student learning (but larger changes impact learning more). For instance, reducing average teacher learning CAL math hours by 1 to 2 h per year (1.6 to 3.3 min per week) or 2 to 3 skills per year decreases student performance. Notably, even minimal CAL usage (3 to 4 h per year or on average 5 to 6.7 min per week) leads to statistically significant learning gains. Further, as shown in Table 2, predicted treatment effects by usage level are similar across linear, quadratic, and cubic models—and while predicted gains are smaller when we replace school fixed effects with school-year fixed effects , they are not substantially dampened. This is visually illustrated in Fig. 3: Our linear specification fits the data well within our empirical support, though uncertainty regarding functional form is heightened at more extreme values.^††

Fig. 3.

Binned, linear, cubic residualized regression results. Note: To residualize variables, we first estimate the relationship between each of the outcome and explanatory variables and the student, teacher, school, and year-grade fixed effects, and then remove the portion of variation explained by these fixed effects. Binned dots represent coefficient estimates from Eq. 5.1, and vertical lines represent associated 95% CIs. The linear model is generated from Eq. 1 and cubic from Eq. 5.2.

Table 2.

Predicted treatment effects by usage and model

	Average usage	Recommended usage
Model	(6.6 annual hours)	(18 annual hours)
Linear (main specification)	$+$0.031 SD	$+$0.085 SD
Linear (school-year FE)	$+$0.021 SD	$+$0.058 SD
Quadratic	$+$0.031 SD	$+$0.084 SD
Cubic	$+$0.031 SD	$+$0.076 SD

Note: This table reports the predicted impact of a math teacher increasing CAL usage among other students by 6.6 and 18 h on standardized math learning standardized within year and grade. Linear (Main Specification) applies coefficient estimates from Eq. 1; Cubic applies estimates from Eq. 5.2; Quadratic applies estimates from Eq. 5.2) omitting the cubed term.

We present additional nonlinearity analyses in SI Appendix, section A6, including detailed regression results comparing linear, quadratic, cubic, and log-transformed models. We also show F-tests comparing linear, quadratic, and cubic model fits, as well as AIC and BIC comparisons between linear and log-transformed specifications. In all comparisons, neither the quadratic nor cubic models yield statistically significant improvements in fit, and their predicted effect sizes are not meaningfully different from the linear estimates. AIC and BIC values indicate that the linear model fits the data better than the log-transformed specification. Therefore, while we cannot rule out nonlinearity, we find no evidence that nonlinear estimation is more appropriate than linear within our empirical support.

Another potential concern is that increases (or decreases) in CAL usage reflect mostly total math instruction and practice time changes, rather than the impact of this CAL platform substituting for other forms of practice. In our sample, on average approximately 80% of CAL platform activity occurred during typical school hours of 8am to 3pm. As we do not expect that time spent on math during school varies significantly across years, it seems unlikely, though not impossible, that average changes in CAL usage across years simply reflect changes in total math learning time.

Heterogeneous Treatment Effects.

Fig. 4 presents the results of Eq. 3. Students from higher quartiles of initial math ability benefit more from CAL than lower-quartile students, regardless of whether we measure usage as leave-student-out teacher mean math hours or math skills. Treatment effect estimates rise with baseline MAP performance, from near zero for Q1 to $+$0.013 SD per hour for Q4 students. For Q4 students, this corresponds to $+$0.241 SD for 30 min of weekly usage or $+$0.088 SD for our observed average of 6.6 h per year spent on CAL. Estimates are more precise when skills mastered (instead of hours practiced) are modeled. Estimates are lower when using skills as the treatment because most students complete multiple skills per hour, with a teacher-level median of 2.60. Importantly, when using leave-student-out skills to quantify CAL usage, all students (including Q1 students) have a statistically significant gain in math learning associated with an additional skill progressed. These findings suggest that while no students are harmed by CAL, higher-performing students see the largest gains.

Fig. 4.

(A) Heterogeneous treatment effects by baseline math performance quartile. (B) Heterogeneous treatment effects by teacher mean skills per hour. Note: Figures report heterogeneous treatment effects ($\beta +\eta$ in Eq. 3). Dots represent coefficient estimates, vertical lines indicate 95% CIs, and horizontal lines show the average treatment effect. SEs are two-way clustered at the math teacher and school level.

We then explore mechanisms: Are students in higher quartile achieving additional gains predominantly due to greater CAL usage? SI Appendix, Table A13 presents estimates of student CAL usage as defined in Eqs. 4.1 and 4.2. Results show that, in response to each teacher mean hour of practice time, higher-quartile students spend marginally more time on CAL and also master more skills. This may be expected if initially high performing students have a stronger math background, interest, or ability.

However, at least for some quartiles, differences in usage alone cannot fully explain the substantial variation in treatment effects. For example, if quartiles differ only in CAL hours per teacher mean practice hour, implied treatment effects exceed observed effects for quartile one students but fall short for quartile four students (implied estimates lie outside the 95% CIs). Similarly, if quartiles differ only in CAL skills per teacher mean hour, this also fails to fully account for quartile four outcomes (the implied treatment effects are below the lower bound of the 95% CI associated with observed effects). For these quartiles, we can rule out that differences in dosage alone explain all of the observed heterogeneity in learning treatment effects. Additional factors beyond CAL usage must contribute to observed heterogeneity, such as differences in learning retention or ability to transfer learned ideas to external assessments.

Second, students benefit most when their teachers have students that use the platform efficiently (i.e., when students focus on skills progression). The estimated impact of an additional CAL usage hour is $+$0.0026 SD when the teacher’s skills-per-hour rate is below the median, but $+$0.0086 when above the median. That is, the treatment effect increases by $+$0.0060, approximately tripling (95% CI [0.0022, 0.0099]) for teachers whose average student skill-mastery-per-hour rate is above the median.

Discussion

We identify the impact of CAL from within-teacher variation in average KA usage over time, controlling for a number of fixed effects. Our average treatment effect estimates^‡‡ suggest that students benefit significantly from KA CAL usage even at low classroom usage levels, as identified using our presented analysis of a large panel dataset. In our panel data, with the average student spending just 11 min per week on CAL, the expected effect of such CAL usage is $+$0.031 to $+$0.054 SDs in math learning per year. If teachers’ students use CAL for approximately 30 min per week (18 h per year), as do 10% of students in our panel data, students may achieve gains of $+$0.085 to $+$0.146 SD per year. Our estimates suggest marginal changes in CAL usage over time—whether teachers increase or decrease their usage—influence student math learning outcomes. If teachers can encourage greater usage, the impact on student learning could grow substantially. Just as important as effect size is the context in which we observe such results: We find these effects in “the real world” where implementation may be imperfect, school and teacher buy-in may be limited, and little to no additional resources and supports are provided. That we still observe positive effects under such conditions highlights the potential of CAL as an affordable, scalable intervention—one that may yield even larger gains if teachers are able to facilitate greater and more effective usage.

Heterogeneous treatment effects reveal important CAL limitations and also point to ways that teachers can potentially unlock greater CAL gains for their students. While CAL improves math learning for all students (even those in the lowest math performance quartile), we find that CAL usage benefits more those students already doing relatively well: Students who already have a strong baseline foundation in math experience the largest returns to CAL usage. There was also a notable difference in average treatment effect of CAL usage between teachers whose students had higher skill acquisition per hour, and those with lower (beyond teacher, student, school, grade, and year effects). Though our work cannot distinguish whether such differences are due to peer effects, or specific teacher guidance to their students on platform use, the substantial benefits of being with a teacher whose students exhibit higher skill acquisition per hour highlight an important direction for future research: understanding how teachers might encourage higher skill-per-hour usage. Such practices may help support students across the spectrum of baseline performance.

Our administrative data contains limited information about instructional processes, and in particular, how KA usage substitutes or complements teacher instruction. We do observe that the vast majority of usage occurs during school day hours (79% in 2020–2021, 87% in 2021–2022, and 84% in 2022–2023 of student usage of KA was during school hours, which we defined as 8am-4pm.) and, since subject schedules are often set at the school level, KA usage likely occurs during math classroom time, rather than being used to add additional math practice on top of standard teacher instruction. In that sense, KA substitutes for other in-class activities—but not for teacher time: There is no reduction in teacher labor cost or total instructional time, so from that perspective, KA functions as a teacher complement. Further, the platform’s dashboard provides teachers with greater visibility into student progress, which could increase the effectiveness of teacher-led instruction during periods when students are not using CAL.

An important and unresolved issue is how teachers modify classroom instruction when they adjust KA time. In separate work, a RAND teacher survey (36) found that about a quarter of teachers using KA used no other CAL platforms.^§§ This suggests, but does not confirm, that some teachers in our sample used KA only; hence, changes in KA usage may directly reflect changes in total CAL usage for those teachers. However, the majority of KA users also report using additional CAL tools, implying that KA is often part of a broader instructional ecosystem. Our small sample of surveyed and interviewed teachers (SI Appendix, sections A8 and A9) further supports this, indicating that shifts in KA usage were frequently due to substitution with other CAL tools and alternative forms of in-class math practice.^¶¶

Even for teachers who use multiple CAL tools, changes in KA usage likely do not translate one-for-one into changes in total CAL practice time. KA usage does not fill available instruction time, so increases in KA usage does not necessarily reduce or eliminate other CAL usage. Similarly, our survey and interview evidence suggest reductions in KA usage can be due to spending that time both on other CAL and on non-CAL practice.

To the extent that KA substitutes directly for other CAL tools, our estimates may understate the total effect of CAL relative to no CAL use. On the other hand, if KA usage moves in tandem with other CAL tools—as complements—our estimates could overstate the marginal effect of KA. However, our interview and survey data suggest that this kind of complementary movement is relatively rare. Notably, our estimated impacts of KA practice time on math learning are similar between teachers who increase KA usage ($+$0.0034 $\pm$ 0.0029) or decrease KA usage ($+$ 0.0043 $\pm$ 0.0045) across years (SI Appendix, section A5.1). This symmetry offers reassuring evidence about the robustness of our estimates, particularly given that the underlying motivations for increasing versus decreasing platform use are likely to differ across instructional contexts.

In summary, our findings suggest that even minor increases in (this particular) CAL platform (While our study focuses on one CAL platform (Khan Academy), our findings are likely generalizable to other platforms that incorporate baseline diagnostic assessment, promote mastery-based learning, and provide immediate feedback to students.) leads to meaningful learning gains; further research is needed to better understand the underlying mechanisms, which will help guide how such gains can be broadly realized.

Supplementary Material

Appendix 01 (PDF)

Data, Materials, and Software Availability

Code to replicate analyses has been deposited in Zenodo (https://doi.org/10.5281/zenodo.17810869) (37). The data are proprietary from NWEA (now bought by Houghton Mifflin Harcourt) and Khan Academy. While the data will not be publicly released, interested parties may contact Bogdan Yamkovenko at bogdankhanacademy.org to inquire about access.