Sector search strategies for odor trail tracking

Significance

Surface-bound odor trail tracking is critical for the survival of terrestrial animals dependent on olfaction. Little is known about how animals track trails at the algorithmic level. In the present study, we propose that a tracking animal maintains a noisy estimate of where the trail is headed based on its past contacts with the trail. We show that virtual agents trained to exploit this strategy reproduce the tracking patterns of ants and rodents. The observed patterns emerge simply as a consequence of common geometric constraints, which also impose fundamental limits on how quickly an animal can track trails. A series of experiments is proposed to quantify how past experience and trail statistics shape tracking behavior.

Abstract

Ants, mice, and dogs often use surface-bound scent trails to establish navigation routes or to find food and mates, yet their tracking strategies remain poorly understood. Chemotaxis-based strategies cannot explain casting, a characteristic sequence of wide oscillations with increasing amplitude performed upon sustained loss of contact with the trail. We propose that tracking animals have an intrinsic, geometric notion of continuity, allowing them to exploit past contacts with the trail to form an estimate of where it is headed. This estimate and its uncertainty form an angular sector, and the emergent search patterns resemble a “sector search.” Reinforcement learning agents trained to execute a sector search recapitulate the various phases of experimentally observed tracking behavior. We use ideas from polymer physics to formulate a statistical description of trails and show that search geometry imposes basic limits on how quickly animals can track trails. By formulating trail tracking as a Bellman-type sequential optimization problem, we quantify the geometric elements of optimal sector search strategy, effectively explaining why and when casting is necessary. We propose a set of experiments to infer how tracking animals acquire, integrate, and respond to past information on the tracked trail. More generally, we define navigational strategies relevant for animals and biomimetic robots and formulate trail tracking as a behavioral paradigm for learning, memory, and planning.

Experimental studies demonstrate the ability of ants, dogs, humans, and rodents to track odor trails (1–6). Rodents accurately track trails in the dark, remaining close to the trail and casting when contact is lost (Fig. 1A) (5). Carpenter ants closely follow a trail while sampling it using a “crisscross” pattern with their two antennae (Fig. 1B) (1). Current models of this behavior rely on variants of chemotaxis (7) based on continuous estimates of the rising and falling odor gradients as the trail is crossed. One such strategy compares simultaneous odor concentrations detected by two spatially separated sensors (8). Yet, rats with a blocked nostril (5) and ants with a single antenna (1) are still able to track trails, although less accurately. An alternative chemotaxis strategy has the animal measuring odor gradients along its trajectory and turning when a significant decrease is perceived (5).

Fig. 1.

Sample trail-tracking trajectories from previous experiments and our RL simulations. (A) A rat (head position in red) tracking a trail (in black). Note the wide casts on extended loss of contact with the trail. Data reproduced from ref. 5. (B) A carpenter ant tracking an odor trail (black) using a stereotyped crisscrossing strategy (1). (C and D) Sample trajectories obtained from RL for agents with one sensor (C) and two sensors (D) recapitulate experimentally observed tracking patterns in A and B. (E, Left) Search paths executed by RL agents with a single sensor upon loss of contact with the trail. (E, Right) The initial prior distribution (E, Bottom Right) over trail headings transforms into a bimodal posterior distribution (E, Top Rightand E, Middle Right), which drives the oscillatory pattern of casting. (F) RL agents with two sensors show a characteristic crisscrossing pattern close to the last detection point. The search path is similar to the single-sensor agent at long distances (SI Appendix, Fig. S1A). (G) RL agents show a trade-off between tracking speed (rescaled by the sector angle σ, sensor size a, and sampling frequency ω) and the probability of losing the trail entirely.

While chemotaxis-based strategies can allow for trail tracking when trails are continuous, they fail when trails are broken and gradients are absent, which is certainly relevant for animals tracking trails in the wild. In experiments with broken trails (1, 5), the absence of signal triggers casting, which is a fundamental feature shared with olfactory searches in a turbulent medium (9–11). Even though turbulent searches also feature sporadic cues, airborne odor signals tend to be localized in a cone, and even within the cone, the signal is highly fluctuating (12, 13). Therefore, beyond qualitative similarities between terrestrial trail tracking and airborne olfactory searches, the specific statistics of detections, geometric constraints, and behavioral patterns are distinct.

In contrast with chemotaxis-based algorithms, we propose an alternative framework built on the searcher exploiting past contacts with the trail to maintain an estimate of the trail’s local heading and its uncertainty. A minimal memory of the approximate locations of the two most recent contacts suffices to delineate an angular sector of probable trail headings that radiates from the most recent detection point. The resulting “sector search” provides a quantitative description of trail-tracking behavior that unifies its various phases and yields specific experimental predictions.

Results

We first show that reinforcement learning (RL) based on the sector search idea can recapitulate natural behavior. An RL agent in this scheme learns to traverse the trail as quickly as possible while minimizing the probability of losing it (Materials and Methods has details). Our in silico RL experiments show that general aspects of animal tracking behavior naturally emerge (Fig. 1 C and D). Specifically, casts are observed around the most likely heading of the trail, and their amplitude is within the angular sector defined by the initial uncertainty σ of the trail’s heading $\varphi$. The reason for the oscillatory pattern of casting is intuitive. Indeed, while moving along a path $\mathcal{C}$ without detecting the trail, the estimated heading’s probability distribution $P(\varphi )$ (Fig. 1E) is updated into $P_{\mathcal{C}}(\varphi )$ as

$$P(\varphi )\to P_{\mathcal{C}}(\varphi )\propto {\mathrm{\Gamma }}_{\mathcal{C}}(\varphi )P(\varphi )\hspace{0.17em}$$ [1]

where ${\mathrm{\Gamma }}_{\mathcal{C}}(\varphi )$ is the probability of not detecting the trail headed along $\varphi$. Irrespective of the explicit form of ${\mathrm{\Gamma }}_{\mathcal{C}}(\varphi )$, the depletion of headings already explored generally leads to a bimodal posterior distribution, with the two modes at the edges of the angular sector (Fig. 1E). The search process is analogous to an agent “foraging” for the trail at two spatially separated patches. The emergence of oscillations is then understood in terms of marginal value theory (14, 15); we show using a minimal model of casting (Materials and Methods) that the turning point of a cast occurs when the marginal value of continuing on one side of the sector (i.e., without paying the cost of traveling) is outweighed by the probability of finding the trail on the opposite side.

We proceed now by establishing geometric limits on tracking speed. A typical RL curve for the probability of losing the trail vs. speed is shown in Fig. 1G. Its monotonicity epitomizes universal limits that “staying on the trail” imposes on tracking speed. Intuitively, searching slowly reduces the distance between detections (the interdetection interval [IDI]), decreasing the uncertainty in the estimate of the trail’s heading and thus, the probability of losing the trail. However, these benefits come at the cost of slow forward progression along the trail. In contrast, moving quickly reduces the detection rate, leading to longer IDIs, increased uncertainty, and loss probability.

We quantify the above trade-off using simple scaling arguments. Suppose the tracking agent has a sensor of size a, samples at a frequency ω, and moves with a fixed forward speed v. As shown in Fig. 2A, the angle subtended by the detector at distance r from the last contact is a / r. The agent searching over an angular sector then scans at a rate $d\varphi /dt=\omega a/r\simeq \omega a/vt$. Integrating the above expression for the angular rate, $\frac{v}{a\omega }{\displaystyle \int d}\varphi ={\displaystyle \int d}t/t$, we obtain the typical time for searching over a sector angle σ: $t_c\sim {\omega }^{-1}{e}^{\sigma v/a\omega }$. The corresponding distance $L\sim v{t}_c$ is obtained using $r\sim vt$. The heading of the trail is known with uncertainty σ, which is the opening angle of the conical sector shown in Fig. 2A. Uncertainty is expected to depend on the distance $L\prime$ from the previous detection as $\sigma (L\prime )={(L\prime /\ell )}^{\gamma }$, where $\ell$ and γ characterize the statistics of trails (below and Fig. 2D). Importantly, a stable strategy for long-term tracking requires that successive IDIs should on average be equal (i.e., $L=L\prime$). Combining L = vt_c with $L\prime ={\sigma }^{1/\gamma }\ell$ and the expression for t_c, we finally obtain an upper bound on the tracking speed v:

$$\frac{v^{1+\gamma }}{a{\ell }^{\gamma }{\omega }^{1+\gamma }}={(\omega t_c)}^{-\gamma }\log (\omega t_c)\le {\gamma }^{-1}{e}^{-1}.$$ [2]

Fig. 2.

History dependence and trail models. (A and B) Trail tracking naturally splits into distinct episodes punctuated by trail detections by the searcher. In each episode, we propose that the tracker searches for the trail using an estimate of the trail’s heading updated based on the past points of contact with the trail and a model of trail statistics. We affix a polar coordinate system with the origin at the most recent contact point and the azimuthal angle defined relative to the estimated trail heading. The uncertainty σ fixes the angular width of the search. The searcher moves forward with a speed v while sampling at a frequency ω. A sensor of size a spans a / r radians at distance r, which determines the rate at which the angular space is searched. (C) To estimate where the trail is headed and its uncertainty from past contacts, the tracker can either use local anisotropy estimated from a single contact (C, Left) or extrapolate from previous points of contact using a model of trail statistics (C, Center and C, Right). In the latter case, the most likely trail paths (dashed blue lines) are similar to interpolated splines, which capture basic geometric notions of persistence in heading and curvature. (D) The uncertainty in trail heading (in radians) as a function of the distance, L, between points of contact for the GWLC model of trails discussed in the text. λ is the correlation length scale of the trail’s curvature. SI Appendix, Fig. S3C illustrates the various scaling regimes exhibited by the GWLC model. (E) The correlation between trail heading at the most recent and second most recent points of contact for the GWLC model changes with the distance between these points, yet it is generally expected to be negative. (F) The expected search distance, vt_c, against the distance, L, between the previous two points of contact for $a/\lambda =0.1$ and $v/a\omega =5,6.25$ (black and gray, respectively) (discussion above Eq. 2). The point of intersection with the ${45}^{°}$ dashed line is the condition for a stable tracking strategy. The gray curve corresponds to $v_{\max }$ beyond which tracking is unstable.

Its maximum $v_{\max }\sim \omega {(a{\ell }^{\gamma })}^{\frac{1}{1+\gamma }}$ defines the optimal stable tracking speed in terms of the tracker’s sensory parameters and trail statistics. The basic element that leads to this bound is the geometric factor $1/r$ that underlies searching over an angular sector. The result from Eq. 2 that $\omega t_c$ is of order one ($e^{1/\gamma }$) explains experimental observations (Fig. 1) that tracking animals typically take only a few samples to reestablish contact with the trail.

The above argument implies that tracking speed depends on the trail statistics via the relation between uncertainty and the distance between points of contact. We use ideas from polymer physics to quantify how this relationship depends on geometric properties of the trails. Specifically, we ask how detecting the trail at a set of points ${\mathit{r}}_0,{\mathit{r}}_1,{\mathit{r}}_2,\dots$ (Fig. 2B) constrains the searcher’s estimate of its future heading. We consider the case when the searcher keeps track of the two most recent points of contact with perfect memory of their location. A more extended memory is discussed further below; an imperfect memory adds to the uncertainty and can be easily accommodated within the framework developed below. Intuition for the two-point case is provided by the familiar “curve” tool in graphical design software, which draws a cubic spline through a set of prescribed points (Fig. 2C). The tool captures the simple intuition that tangents to a curve are continuous (i.e., the trail’s heading has local persistence), which is a plausible, minimal assumption about trails. We show in SI Appendix that cubic spline interpolation corresponds to the most likely path (through a fixed set of points) in the so-called worm-like chain (WLC) ensemble (originally introduced for polymers) (16, 17). In this ensemble, the tangent direction undergoes diffusion with rate κ, and the uncertainty is then $\sigma ={(\kappa L/3)}^{\frac{1}{2}}$, which determines the two parameters: the scaling law, $\gamma =1/2$, and the correlation length scale, $\ell =3{\kappa }^{-1}$, in Eq. 2. Actual trails could be smoother and have a well-defined curvature (the rate of change of heading) that persists on a characteristic length scale λ. We capture this ensemble of curves by introducing two additional parameters: persistence length λ and typical radius of curvature ξ (Materials and Methods). Uncertainty is then given by $\sigma \approx L/2\xi$ (hence, γ = 1 and $\ell =2\xi$) at distances $L<\lambda$, while diffusive scaling is recovered at larger distances with an effective diffusivity $\kappa =2\lambda {\xi }^{-2}$. This extended model defines a generalized worm-like chain (GWLC) ensemble with cross-overs across the various regimes (Materials and Methods). In summary, the model leads to a “propagator,” which encodes how information about past contacts is integrated to form an estimate of the trail’s heading while taking into account geometric aspects of trails. A general feature is that the headings at two consecutive contacts are anticorrelated (Fig. 2E), which reflects the bending of the spline relative to the chord seen in Fig. 2C. We emphasize that although the general strategy of the agent depends on the statistical properties of the trail ensemble, the specific actions taken by the tracking agent along a particular trail, such as reorientation based on the most likely trail heading, will depend on the history of contact points via the propagator for the WLC (or GWLC) model.

Why and when do searchers need to cast? The question stems from our previous result that a few samples are typically sufficient to reestablish contact with the trail. To address it quantitatively, we consider again the setup of Eqs. 1 and 2. The nondetection probability averaged over the ensemble of trails that pass through past contact points is

$${\mathrm{\Gamma }}_{\mathcal{C}}={\langle e^{-\omega {\displaystyle {\int }_{\mathcal{C}}}\frac{ds}{v}{I}_a\left(\mathit{r}\left(s\right),\mathit{y}\left(s\right)\right)}\rangle }_{\mathit{y}},$$ [3]

where s parametrizes the searcher’s path $\mathcal{C}$ and the Boolean indicator function I_a measures if the agent at $\mathit{r}(s)$ is within sensing range a of the trail at y (i.e., the integral is the time spent in contact with the trail). Numerical simulations of the search show a power law scaling regime for ${\mathrm{\Gamma }}_{\mathcal{C}}$, which is cut off at short distances by the initial surge and at long distances by trails escaping out of the casting envelope (Fig. 3 A–C). We proceed to explain these three regimes shown in Fig. 3B. Intuitively, at short radial distances $r<a/\sigma \sim v/\omega$ (the latter from Eq. 2), the sensor covers the entire sector of likely headings, the searcher can just move forward, and ${\mathrm{\Gamma }}_{\mathcal{C}}\propto e^{-\omega r/v}$ (Fig. 3B). Casting sets in if the searcher reaches, without detection, a distance $r\mathrm{\gtrsim }a/\sigma$ (i.e., when the sector is not fully covered any longer by the sensor of size a). The sector geometry in Fig. 3A implies that the length of a single casting sweep is proportional to r. A fixed forward speed then implies that the distance between successive encounters with the trail also scales with r. Hence, the number of times the searcher crosses the trail (and thus, the time spent on the trail) per unit radial distance decreases as $1/r$. This $1/r$ scaling in the overlap then leads to a logarithmic integral in the exponent of Eq. 3 and thus, a power law regime ${\mathrm{\Gamma }}_{\mathcal{C}}\propto r^{-\beta }$ during casting. The optimal exponent ${\beta }^{*}$ depends on the statistics of the trails, yet it generally satisfies ${\beta }^{*}>1$ (${\beta }^{*}=1.63$ for the curvature-dominated GWLC model) (Eq. 19). Since ${\mathrm{\Gamma }}_{\mathcal{C}}$ is a cumulative distribution, $\beta >1$ implies that the mean distance is indeed determined by the lower cutoff (i.e., the trail is typically found in a few samples, as estimated in Eq. 2). However, the power law decay implies that casting phases are frequent and can span up to the upper cutoff where all aspects of trail statistics and search geometry come into play, as discussed below.

Fig. 3.

The role of casting in trail tracking. (A–C) The probability of not detecting the trail at distance L decays exponentially during the initial forward surge (red) and as a power law during casting (blue) over a conical envelope. The searcher often finds the trail within the initial surge (mean in B), whereas casting determines the probability of losing the trail (determined by the upper cutoff) if the trail is not found during the surge. Beyond a characteristic length scale $\ell$, the detection rate becomes negligible (green region in B) as trails initially well inside the casting envelope escape out of the envelope (C). (D and E) The casting policy obtained from Bellman optimization. The specific casting strategy depends on trail statistics, shown here for diffusive and curvature-dominated trail ensembles (scaling exponent $\gamma =0.5,1.0$) in blue and red, respectively. Note the increasing casting angle and the slowing down of the agent at $r\approx \ell$ (E). (F) The trade-off between the probability of missing the trail for fixed $\langle L\rangle$ and searcher speed v.

How should an agent perform sustained casts so as to minimize the probability of losing the trail while maximizing tracking speed? To go beyond the above scaling arguments, we now consider the geometry of sector search in detail. The searcher’s path is parameterized by the sequence of turning points of its casting trajectory $\{r_k,{\theta }_k\}$ (in polar coordinates with respect to the most recent point of contact), which are to be optimized. We maximize the average tracking speed L / T, where L and T are the distance and duration between the most recent and the subsequent point of contact with the trail, respectively. As discussed previously, the uncertainty estimated for a bout of sector search depends on the IDI. To constrain the uncertainty, we therefore constrain the average, $\langle L\rangle$. Hence, we consider the following optimization problem:

$$V_{v,\mathrm{\Lambda }}={\max }_{\{r_k,{\theta }_k\}}\left[\langle \frac{L}{T}\rangle -\mathrm{\Lambda }\langle L\rangle \right],$$ [4]

where Λ is the Lagrange multiplier enforcing the $\langle L\rangle$ constraint. The turning points and the searcher’s speed v affect the probability of detecting the trail in a single cast, which is implicit in the expectation in Eq. 4. We solve the Bellman equation corresponding to the above optimization problem using dynamic programming (Materials and Methods), which sequentially optimizes for the turning points by considering at each step the two possibilities that either the trail is detected during the cast or the agent advances to the next cast. The probabilities for these two events are controlled by the nondetection probability given by Eq. 3. In the event of no detection, the estimate of the trail’s heading is updated according to Eq. 1. The resulting optimization yields a search strategy with an increasing sequence of casting angles (Fig. 3D). The specific casting strategy depends on how trails meander and curve (Fig. 3E). The choice of v controls the trade-off between the tracking speed and the probability of losing the trail entirely (Fig. 3F). Independent of the choice of v and $\langle L\rangle$, azimuthal excursions are by and large conical but extend dramatically (Fig. 3 D and E) when trails that were initially inside the cone escape from it with high probability (Fig. 3C). This happens at a distance that scales with $\ell$ but also depends on the sector envelope, which in turn, depends on σ (recall that $\ell$ is the correlation length of the trail heading). Intuitively, at length scales $\sim \ell$, the trail’s heading decorrelates from its initial value, the relevance of information on past detections has expired, the trail is effectively lost, and it is optimal to stop progressing forward.

Experimental Tests

A number of transformative experimental assays are suggested by our theoretical framework. The broad theme is whether and how animals adapt their behavior to the statistics of trails. For field experiments, it would be informative to measure the statistics of natural trails, analogous to the statistics of natural images that has brought insight into the adaptation of visual responses (18–21). Specifically, one can measure the autocorrelation of local heading and curvature of natural trails, which would test the validity and fix the parameters of our WLC-type models. In laboratory settings, the statistics of trails can be controlled by varying persistence or curvature or using broken trails (Fig. 4A). The general issue of adaptation is articulated in the following four specific questions that stem from our work.

Fig. 4.

Behavioral assays to dissect trail-tracking strategies. (A) A tracker executing a sector search often maintains continuous contact with the trail (A, Left). Extended sector searches can instead be systematically elicited using broken trails (A, Right). The subsequent search envelope’s orientation and width relative to past contact points inform the tracker’s internal estimate and uncertainty σ of the trail’s heading. (B) We propose dashed trails as an assay to infer how a tracking animal integrates past information. The distance between dashes forces contact points to be separated by at least δL, and the subsequent search sector yields an estimate of σ. (C) Our theory predicts that the most likely trail heading ${\varphi }_{\text{ML}}$ is proportional to the angle ψ between the line segments joining points of contact (red dots), with a prefactor that depends on the distance δL between the two most recent contact points, the trail model and memory. (D) Automated behavioral tracking of rodents on a treadmill allows control of tracking speed and trail statistics (5, 22). (E) The mean distance to the trail with speed in simulations of sector search (as in A, Left), which recapitulates the linear relationship (dashed line) observed in experiments with rats (5). We use a sector search strategy, where the longitudinal speed v is fixed and the tracker rapidly casts within a conical envelope (SI Appendix, Fig. S4A shows a sample trajectory in a single bout). For each v, we simulate 100 trials, where each trial consists of successive 10 successful contacts with the trail. Trails are generated from the GWLC ensemble with $\kappa =0,\xi /\lambda =3,a/\lambda =0.1$. Error bars are one SEM.

First, how long after the loss of contact do animals “give up” tracking? Our prediction is that they should when they get beyond the characteristic correlation length of the trails. At this point, the value of past information has expired, and it is best to turn back or start a new search. This prediction can be tested by varying trail statistics, interrupting the trails, and measuring when animals give up.

Second, does the amplitude of casting depend on the statistics of trails and the IDI? We predict that it should, and the specific quantitative relationship is a signature of the underlying predictive model employed by the animal (Fig. 4B). Our prediction should be contrasted with the nonadaptive casting envelope assumed in ref. 5. The IDI can be experimentally manipulated by generating dashed trails as shown in Fig. 4B, which forces the animal to detect the trail sporadically yet at controllable intervals. It would be particularly informative to verify whether or not animals include curvature in their estimates of the trails’ future heading or limit to persistence.

Third, what is the memory of past trail contacts? Experiments with forked trails (5) show that rats exhibit a predictive component, suggesting a memory that extends over the recent past. Our theory posits that the tracker remembers (at least) the two most recent detection points. For the case of two-point memory, we predict the sector search is oriented along the line connecting those two points. If more than two points are remembered, the expected heading deviates from this line (by an angle that we calculate explicitly in SI Appendix) as illustrated in Fig. 4C. Note that the heading is not an average of the past headings, as assumed in ref. 5, and actually depends on the IDI between recent contacts; this prediction could again be tested by using curved, dashed trails as in Fig. 4B.

Fourth, does the tracking speed vary with the typical IDI, reducing with increased uncertainty as predicted by the speed–accuracy trade-off Eq. 2? This can be tested by varying the speed, for instance, of the treadmill in ref. 5 (Fig. 4D) and measuring tracking accuracy. Available data for three speeds in ref. 5 are captured by our theory (Fig. 4E), which highlights the importance of revisiting those pioneering experiments and measuring additional quantities, namely the explicit prediction in Fig. 3F.

Conclusion

In conclusion, we show that an optimized sector search strategy based on the memory of two or more recent detection events yields an oscillatory search path with increasing amplitude that naturally unifies the observed low-amplitude “zigzagging” and larger-amplitude “casting” behaviors into the same quantitative framework. This framework elucidates the geometric and computational constraints faced by tracking animals and identifies general features of the algorithms that efficiently solve the task, which can also be implemented for robotic applications. Insights and predictions developed here impact and should inform the design and analysis of future animal behavior experiments.

While the computational constraints discussed in this work apply generally, animals may face an additional set of species-specific physical and physiological constraints that influence aspects of their trail-tracking strategies. For example, inertial effects and gait constraints will affect the rate at which an animal sweeps across a sector and consequently, its speed selection. Further flexibility is offered by extra degrees of freedom in the sensorimotor apparatus, such as modulation of the inhalation rate or independent control of segments of an ant’s antennae. In our model, the sensorimotor mechanism impacts the strategy via the probability of detection of the trail from the animal’s sensors, which we have assumed to be fixed. Indeed, without additional trade-offs, our model implies that the optimal strategy is to simply pick the parameters that maximize the probability of detection (i.e., the maximal sensor size and sampling frequency). Physiological constraints may impose trade-offs (e.g., between the strength and rate of inhalation), which then lead to an altogether new dimension in the animal’s decision making. We postpone a detailed examination of these trade-offs and how they influence the trail-tracking strategy to future work.

We have assumed that the searcher’s behavior is guided by a model of continuous trails and investigated how an animal should behave in the presence of a sporadic break (a common perturbation in experiments). Recurrent breaks in the trails can be incorporated into the decision-making framework by introducing an additional term in the exponent of Eq. 3 for the presence or absence of a trail. When averaged over the trail ensemble and the statistics of brokenness, the effect of broken trails appears as a reduced detection rate if the breaks are much shorter than the correlation length scale $\ell$, and the general conclusions remain unaffected. However, if the breaks last longer than $\ell$, the trail is “patchy,” with each patch containing little information about the location of the next patch. In such a scenario, a strategy that employs biased random exploration or foraging may prove optimal.

Materials and Methods

An RL Framework for Trail Tracking

Trail tracking naturally splits into discrete episodes where after each loss of contact, the tracker searches and attempts to reestablish contact with the trail. We use RL to identify optimal search strategies for each episode and explore how factors such as sensory configuration or movement constraints influence the strategy. The task is a one-dimensional search over angular space θ, the geometry of which is illustrated in Fig. 2A. The tracker controls its tangential speed $u(t)\equiv r\dot{\theta }$, while its radial speed v is kept constant. For simplicity, we focus here on the configuration featuring a single sensor of size a sampling with a Poisson frequency ω. The generalization to two sensors is found in SI Appendix.

In each episode, the (Bayesian) agent maintains a posterior probability distribution function (PDF) $P(\varphi )$ over possible trail headings, which is continuously updated based on the locations already visited until the trail is recontacted. The agent’s strategy of decisions about its future trajectory a priori depends on the full high-dimensional distribution $P(\varphi )$, which is difficult to learn. To circumvent this issue, we formulate the search task using a tractable parametrization of the posterior as a mixture of K Gaussian basis functions:

$$P(\varphi )={\displaystyle \sum _{i=1}^K{q}_i}{B}_i\left(\varphi \right),\hspace{1em}\hspace{1em}{B}_i(\varphi )=s^{-1}\phi \left(\frac{|\varphi -{\mu }_i|}{s}\right),$$ [5]

where $\phi$ is the standard normal PDF and the q_i ’s are normalized weights. The posterior is encoded by the weights q, the posterior probabilities of the latent states given the agent’s history. The corresponding vector is lower dimensional and yields to standard RL methods. For simulations in the text, we used $K=3$; equal initial weights; ${\mu }_1,{\mu }_2,{\mu }_3=-\sigma ,0,\sigma$; and $s=0.5\sigma$. These values were chosen so that the prior has mean zero and SD $\approx \sigma$. Using K > 3 led to similar strategies of search, but training was slower. We define the detection probability given latent state i for an agent at r as

$${\overline{\gamma }}_i(\mathit{r})={\displaystyle \int \gamma }(\mathit{r},\varphi )B_i\left(\varphi \right)d\varphi ,$$ [6]

where $\gamma (\mathit{r},\varphi )$ is the detection probability of finding a trail headed along $\varphi$ if the searcher is at r. We assume a Gaussian detection kernel of size a, with distance measured to the closest point on the trail: $\gamma (\mathit{r},\varphi )=e^{-r^2{\sin }^2(\theta -\varphi )/2a^2}\approx e^{-r^2{(\theta -\varphi )}^2/2a^2}$, where we have used the small-angle approximation. Conditional on no detection at r, Bayes’ rule yields ${\dot{q}}_i=-\omega q_i({\overline{\gamma }}_i(\mathit{r})-\overline{\gamma }(\mathit{r}))$, where $\overline{\gamma }={\displaystyle {\sum }_i{q}_i}{\overline{\gamma }}_i$ is the total probability of detection. From Eq. 6, we have

$${\overline{\gamma }}_i(\mathit{r})=\sqrt{\frac{\pi }{2}}\frac{{\sigma }_r}{{\sigma }_r\prime }\phi \left(\frac{\theta -{\mu }_i}{{\sigma }_r\prime }\right),$$ [7]

where ${\sigma }_r=a/r$ and ${\sigma }_r\prime =\sqrt{{\sigma }_r^2+s^2}$.

We use a discount rate λ and provide a reward as discussed below, after Eq. 8. Training is performed in an episodic fashion with each episode lasting time T. The kinematic variables are updated with time step dt, and actions are taken with time step $d{t}_{\text{act}}\ge dt$. Movement constraints are imposed by restricting the set of actions to three values, $u/a\omega \in \{-\alpha ,0,\alpha \}$. The state space has four dimensions: r, θ, $-\ln q_1/q_2$, and $-\ln q_3/q_2$. We discretize our state space using a nonoverlapping tile coding scheme (23). We refer to SI Appendix for hyperparameter values, details about the state space architecture, and the case of two sensors.

We use the SARSA (state-action-reward-state-action) Q-learning algorithm (23), which learns the so-called Q function, that is the value function for each action in a given state:

$$\begin{array}{l}{Q}_{\pi }(\mathit{r},\mathit{q},u)=\overline{\gamma }(\mathit{r})dt+\\ \left[1-(\lambda +\overline{\gamma }(\mathit{r}))dt\right]V_{\pi }\left(\mathit{r}+\mathit{v}dt,\mathit{q}+\stackrel{.}{\mathit{q}}dt\right)\hspace{0.17em}\end{array}$$ [8]

where $V_{\pi }(\mathit{r},\mathit{q})={\displaystyle {\sum }_u{Q}_{\pi }}(\mathit{r},\mathit{q},u)\pi (u|\mathit{r},\mathit{q})$, and the index π highlights the dependence on the probabilistic policy $\pi (u|\mathit{r},\mathit{q})$. The above equation differs from the standard SARSA update by the addition of $\overline{\gamma }dt$ in the discount term, which is due to our formulation of the search as a continuing process conditional on no contact with the trail. Alternatively, one may provide a unit reward when the trail is found, stop the episode, and start over. However, the credit assignment problem in goal-oriented tasks makes the training (Eq. 9 hereafter) problematic, even though the final optimal policy is equivalent (e.g., ref. 24). Our formulation circumvents both issues by 1) giving a local reward $\overline{\gamma }dt$ rather than a final one, which addresses the credit assignment, and 2) including the detection probability $\overline{\gamma }dt$ into the discount rate to account for the condition of no contact.

To learn the Q function as defined in Eq. 8, we use a “softmax” training policy: that is, $\ln \pi (u|\mathit{r},\mathit{q})\propto \widehat{Q}(\mathit{r},\mathit{q},u)/T_{\text{explo}}$, where $\widehat{Q}$ is the current estimate of the Q function and $T_{\text{explo}}$ is a “temperature” parameter that is annealed as training progresses to allow for sufficient exploration of actions. Given an action u at state $(\mathit{r},\mathit{q})$ and a subsequent action $u\prime$ at state $(\mathit{r}\prime ,\mathit{q}\prime ),\hspace{0.17em}\widehat{Q}$ is updated during training as

$$\begin{array}{l}{\widehat{Q}}_{\pi }(\mathit{r},\mathit{q},u)\to {\widehat{Q}}_{\pi }(\mathit{r},\mathit{q},u)(1-\eta )+\\ \eta \left(\overline{\gamma }(\mathit{r})dt+(1-(\lambda +\overline{\gamma }(\mathit{r}))dt){\widehat{Q}}_{\pi }(\mathit{r}\prime ,\mathit{q}\prime ,u\prime )\right)\hspace{0.17em}\end{array}$$ [9]

where η is the learning rate. The function $\widehat{Q}$ obtained at the end of the training period yields a search strategy as ${\pi }^{*}(\mathit{r},\mathit{q})=\arg {\max }_u\widehat{Q}(\mathit{r},\mathit{q},u)$.

We applied the algorithm just described for a range of values of $v,\alpha$, and d (half the distance between sensors for the two-sensor case). For each parameter set, we obtain a search strategy, the corresponding probability ${\mathrm{\Gamma }}_{\mathcal{C}}(T)$ of missing the trail in time T, and the expected number of samples to find the trail. Comparing ${\mathrm{\Gamma }}_{\mathcal{C}}(T)$ on a test set for different numbers of training episodes (SI Appendix, Fig. S1B) shows that nontrivial learning takes place, typically saturating at $\sim {10}^4$ training episodes.

Casting in a Minimal Model of Sector Search

In order to establish the relation between casting and marginal value theory, here we propose a minimal model of sector search. The model lends to an analytical solution, which allows us to quantify how the frequency of casting and the efficiency of search depend on the movement and computational constraints imposed upon the tracker.

We consider the same setting as the above episodic RL framework, where the tracker is searching for the trail over a sector after losing contact with it. To focus on casting, we analyze the behavior of the searcher after an initial forward excursion along the most likely heading. This surge decreases the probability weight q of the mode at $\varphi =0$ and yields a symmetric bimodal posterior distribution concentrated at the two modes, $\pm {\varphi }_0$ (typically ${\varphi }_0\sim \sigma$). The resulting model is equivalent to Eq. 5 with K = 2, ${\mu }_1=-{\varphi }_0,{\mu }_2={\varphi }_0$, and s small. The searcher moves radially as r(t) (fixed), controls its tangential speed $u=r\dot{\theta }$, receives a unit reward when it finds the trail, and incurs a movement cost per unit time $\mu u^2/2$, where μ sets the movement constraint. The reward and cost are discounted at a rate λ. The two-dimensional state space of the agent consists of θ and the probability q of finding the target at ${\varphi }_0$ (the probability is $1-q$ at $-{\varphi }_0$). For full details, we refer to SI Appendix.

The above model is exactly solvable. An optimal searcher exhibits oscillations between $-{\varphi }_0$ and ${\varphi }_0$ (casting) until it finds the trail (SI Appendix, Fig. S2A). After an initial transient, the searcher traverses a loop in state space $(\theta ,q)$, alternating between sampling at $-{\varphi }_0$ or ${\varphi }_0$ and casting to the other side:

$$\begin{array}{l}({\varphi }_0,q_s^{*})\underset{}{\overset{\text{cast}}{\to }}(-{\varphi }_0,q_s^{*})\underset{}{\overset{\text{sample}}{\to }}(-{\varphi }_0,1-q_s^{*})\underset{}{\overset{\text{cast}}{\to }}\\ ({\varphi }_0,1-q_s^{*})\underset{}{\overset{\text{sample}}{\to }}({\varphi }_0,q_s^{*}),\end{array}$$ [10]

where $q_s^{*}$ is the optimal switching probability at which the searcher stops sampling at ${\varphi }_0$ and traverses to $-{\varphi }_0$. The speed of traversal from ${\varphi }_0$ to $-{\varphi }_0$ is determined by balancing the cost of traversing quickly and the potential value at $-{\varphi }_0$ discounted due to the limited time horizon.

Intuitively, the searcher casts when the marginal value of continuing to sample at ${\varphi }_0$ is just outweighed by the marginal value that the searcher receives if it stops sampling, traverses from ${\varphi }_0$ to $-{\varphi }_0$, and samples at $-{\varphi }_0$. Balancing the marginal value of these two possibilities then yields the optimal switching probability $q_s^{*}$. For small $q_s^{*}$, we derive (SI Appendix)

$$q_s^{*}\approx \frac{\lambda }{1+\lambda }\frac{{\zeta }^2}{1+\zeta +(1+\lambda ){\zeta }^2-{\zeta }^3},$$ [11]

where $\zeta \equiv 1-\sqrt{2{\varphi }_0^2\mu \lambda (1+\lambda )}>0$. When $\zeta <0$, the movement cost outweighs the value the agent may receive, and the optimal strategy is to simply not move. $q_s^{*}$ get smaller, and thus, the searcher casts less frequently with increasing time horizon ($\lambda \ll 1$) or movement costs ($0<\zeta \ll 1$). The probability of not detecting the trail after time t is given by

$${\mathrm{\Gamma }}_{\mathcal{C}}(t)\simeq e^{-I{\displaystyle {\int }_{t_0}^t\frac{ds}{r(s)}}},$$ [12]

where I is interpreted as the rate of information acquisition. Its dependence on μ, λ, and ${\varphi }_0$ is shown in SI Appendix, Fig. S2B. As expected, increasing movement cost (decreasing ζ) decreases how quickly information is acquired. Similarly, a large time horizon (λ small) makes the agent sample at $\pm {\varphi }_0$ longer and cast slower, decreasing the rate of information acquisition. For a constant radial speed v, the probability ${\mathrm{\Gamma }}_{\mathcal{C}}(t)\simeq t^{-I/v}$ decreases as a power law, which arises quite generally from the sector search geometry as discussed in the text.

Trail Statistics

If an agent makes contact with the trail at two points separated by distance L, statistical and geometric information is encoded in the propagator $P({\varphi }_L,{\varphi }_0)$, where ${\varphi }_L,{\varphi }_0$ are the trail headings at the two points (measured relative to the line joining them). If H points of contact are remembered, then the propagator can be used to compute the posterior distribution of the heading: $P(\varphi )={\displaystyle \int \mathcal{D}}{\varphi }_{i}P(\varphi |{\varphi }_1)P({\varphi }_1|{\varphi }_2)\dots P({\varphi }_{H-2},{\varphi }_{H-1})$. We introduce an ensemble of trails (which we call the GWLC ensemble) that have persistence in heading and curvature, quantified respectively by the parameters κ and λ (distinct from the discount rate used for RL) and a typical radius of curvature, ξ. Typical samples of trails from this ensemble are presented in SI Appendix, Fig. S3A. The WLC ensemble (16, 17) previously introduced for polymers is a special case with $\xi =\infty$.

The propagator $P({\varphi }_L,{\varphi }_0)$ takes into account all the trails constrained to pass through two contact points, weighted by their probability. We affix a coordinate system, where the line joining the two contact points is along the x axis. Specifically, the trail’s y coordinate, y_x, satisfies the constraint $y_0=y_L=0$; the small-angle approximation is used so that ${\dot{y}}_x={\varphi }_x$; and the trail has end-point headings ${\varphi }_0,{\varphi }_L$. We define

$$P({\varphi }_L,{\varphi }_0)=Z^{-1}{\displaystyle \int \mathcal{D}}{\varphi }_x\mathcal{D}{\chi }_x{e}^{-\mathcal{E}(\{{\varphi }_x,{\chi }_x\})},$$ [13]

where Z is a normalization constant, and the integral is over all possible headings and curvatures ${\varphi }_x,{\chi }_x$ at the various positions x. The action $\mathcal{E}$ of a path is given by

$$\begin{array}{l}\mathcal{E}(\{{\varphi }_x,{\chi }_x\})=\frac{1}{2\kappa }{\displaystyle {\int }_0^{L}d}x{({\dot{\varphi }}_x-{\chi }_x)}^2\\ \hspace{1em}+\frac{\lambda {\xi }^2}{4}{\displaystyle {\int }_0^{L}d}x{\left({\dot{\chi }}_x+\frac{{\chi }_x}{\lambda }\right)}^2+\frac{{\xi }^2{\chi }_0^2}{2}.\end{array}$$ [14]

The model Eq. 14 is a Gaussian process. Since symmetry dictates that $\langle {\varphi }_L\rangle =\langle {\varphi }_0\rangle =0$, it follows that $P({\varphi }_L,{\varphi }_0)$ is defined entirely by the variance ${\sigma }^2(L)\equiv \langle {\varphi }_L^2\rangle =\langle {\varphi }_0^2\rangle$ and by the correlation $\rho (L)\equiv \langle {\varphi }_0{\varphi }_L\rangle /{\sigma }_L^2$.

We present the full calculation of $\sigma (L)$ and $\rho (L)$ in SI Appendix. In summary, the first integral over χ_x can be performed using the Gaussian integral formula and leads to an effective action in ${\varphi }_x$. The Euler–Lagrange equation of this effective action then yields extremal paths (“splines”) that minimize the effective action (SI Appendix has details). The splines have the form

$$\begin{array}{l}{y}_x=a_0(L-x)+a_{L}x+c_0\frac{{(L-x)}^3}{6}+c_L\frac{x^3}{6}\\ \hspace{1em}+{\lambda }^2{d}_0{e}^{-(L-x)/\lambda }+{\lambda }^2{d}_L{e}^{-x/\lambda },\end{array}$$ [15]

where the constants in the above equation are set so that $y_0=y_L=0$ and can be expressed in terms of ${\varphi }_0,{\varphi }_L$. SI Appendix, Fig. S3B shows the splines between contact points spaced at increasing intervals. Plugging Eq. 15 in the effective action, we obtain

$${\sigma }^2(L)=\frac{\kappa {\mathrm{\Omega }}^2{\lambda }^2}{L}\frac{b_1+b_2}{2},\hspace{1em}\rho (L)=\frac{b_2-b_1}{b_2+b_1},$$ [16]

where the two functions b₁ and b₂ are

$$\begin{array}{l}{b}_1=\frac{L^2}{2}-\frac{\lambda L}{2{\mathrm{\Omega }}^2{V}^2}\left(1-e^{-\frac{L}{\lambda }}\right)\hspace{0.17em}\\ b_2=\frac{L^2}{6}-\frac{\lambda (L+2\lambda )}{2{\mathrm{\Omega }}^2{V}^{2}L}\left[\left(L-2\lambda \right)+e^{-\frac{L}{\lambda }}\left(L+2\lambda \right)\right],\end{array}$$ [17]

and $V=\sqrt{\kappa {\xi }^2\lambda /2},{\mathrm{\Omega }}^2={\lambda }^{-2}+V^{-2}$. The variance $\sigma (L)$ and the correlation $\rho (L)$ are plotted in SI Appendix, Fig. S3 C and D. Three distinct regimes are apparent. For $L/\lambda \ll 1$, we can appproximate ${\sigma }^2(L)\approx \kappa L/3+L^2/4{\xi }^2$, which reflects diffusive $\propto L$ and curvature-dominated $\propto L^2$ scalings. When diffusion dominates, $\rho (L)=-1/2$, whereas $\rho (L)\approx -1$ when curvature dominates. The perfect anticorrelation in the curvature-dominated regime is intuitive as the line joining the two points of contact can be viewed as the chord of a circle with radius ξ, whereas $\sigma (L)=L/2\xi$ is the angular deviation of the trail around this chord. When $L/\lambda \gg 1$, the heading is randomized over many correlation lengths, and the diffusive scaling ${\sigma }^2(L)\approx 2\lambda L/3{\xi }^2$ is recovered.

While $P({\varphi }_L|{\varphi }_0)$ (the interpolation model) is required to integrate past information, search strategies also require the forward propagator $P({\varphi }_L,y_L|{\varphi }_0)$, which keeps track of trail headings and locations while the agent searches for the trail. The methods described above can be used again (SI Appendix has details) to yield extremal paths as Eq. 15 and the three quantities ${\langle {\varphi }_L^2\rangle }_{\text{fwd}},{\langle y_L^2\rangle }_{\text{fwd}},{\langle {\varphi }_L{y}_L\rangle }_{\text{fwd}}$. These quantities fully describe $P({\varphi }_L,y_L|{\varphi }_0)$. We validated our interpolation model Eq. 16 and the forward model using numerical simulations (SI Appendix, Fig. S3 E and F).

The Nondetection Probability during Surge and Cast

We introduce a sector search strategy that allows us to quantify the nondetection probability taking into account the full dynamics of the trails and yields intuition on the factors that contribute toward losing the trail entirely. We suppose the radial speed v is fixed and the tangential speed $u\gg a\omega$. In other words, the agent casts rapidly within a conical envelope of semiaperture angle $\sigma {\mathrm{\Theta }}_0$, where σ is the prior uncertainty of the trail’s heading and ${\mathrm{\Theta }}_0\mathrm{\gtrsim }1$ (SI Appendix, Fig. S4A). To simplify the presentation, we assume curvature-dominated trails [i.e., $\sigma (L)=L/2\xi$], although the arguments below are general.

The rapid casting limit $u/a\omega \gg 1$ allows us to compute the nondetection probability at distance r from the most recent contact point defined by Eq. 3:

$${\mathrm{\Gamma }}_{\mathcal{C}}(r)={\langle e^{-\frac{\omega }{v}{\displaystyle {\int }_0^{r}d}x\gamma (x,y(x))}\rangle }_{y(x)},$$ [18]

where the expectation is over the full ensemble of trails $\{y(x)\}$ that pass through past detection points. Here, we provide intuition for the detection rate $\omega \gamma (x,y(x))$ and refer to SI Appendix for full details. For small x (i.e., $x\mathrm{\lesssim }a/2\sigma {\mathrm{\Theta }}_0$), $\gamma (x,y(x))=1$ for most trails as the sensor size a spans the entire casting envelope $2\sigma x{\mathrm{\Theta }}_0$. In the casting regime ($x\mathrm{\gtrsim }a/2\sigma {\mathrm{\Theta }}_0$), since the time spent on the trail in a single cast is a / u, the probability of nondetection per crossing is $e^{-a\omega /u}$. After n crossings, the nondetection probability is then $e^{-na\omega /u}$. As the agent moves a distance dx in the radial direction, it crosses the trail $n=\mathit{udx}/2v\sigma x{\mathrm{\Theta }}_0$ times [i.e., $\gamma (x,y(x))=a/2\sigma x{\mathrm{\Theta }}_0$ ], which is independent of u. From Eq. 18, these relations yield an exponential ${\mathrm{\Gamma }}_{\mathcal{C}}(r)$ during the initial surge followed by a power law, $r^{-\beta }$ with exponent $\beta \equiv a\omega /2\sigma v{\mathrm{\Theta }}_0$. This heuristic argument aligns well with the results of numerical simulations in SI Appendix, Fig. S4B.

The spline formulation of the GWLC yields a geometric picture of the dynamics. Intuitively, the nondetection probability (and thus, the posterior probability from Eq. 1) is large if the trail has a large probability of escaping the casting envelope before it is found by the tracker. In order to “escape” the casting envelope, the trails that are initially well within the casting envelope have to bend significantly. This bending incurs a cost in the action Eq. 14, which reduces with increasing r. The nondetection probability conditional on initial trail heading, ${\mathrm{\Gamma }}_{\mathcal{C}}(r|\varphi )$, flattens out (i.e., the detection rate vanishes) when a significant fraction of trails escapes the casting envelope, including trails with initial heading $\varphi =0$ (Fig. 3C).

This geometric picture yields a length scale $L_{\text{in}}$, which is the distance at which the trails initially along the most likely heading $\varphi =0$ escape the casting envelope. For curvature-dominated trails, escaping trails should deviate by an angle $\sigma {\mathrm{\Theta }}_0\simeq L_{\text{in}}/2\xi$, which gives $L_{\text{in}}\simeq 2\sigma \xi {\mathrm{\Theta }}_0$. The probability of losing the trail can then be estimated as ${\mathrm{ϵ}}_{\text{in}}={\mathrm{\Gamma }}_{\mathcal{C}}(L_{\text{in}})$. An additional contribution to the flattening of ${\mathrm{\Gamma }}_{\mathcal{C}}(r)$ comes from the trails that are headed outside of the cone $\left|\varphi \right|>{\mathrm{\Theta }}_0$ even before reaching $L_{\text{in}}$. The probability ${\mathrm{\Gamma }}_{\mathcal{C}}(r)$ would then flatten out at a different length scale $L_{\text{out}}$, where ${\mathrm{\Gamma }}_{\mathcal{C}}(L_{\text{out}})={\mathrm{ϵ}}_{\text{out}}$. The relative contributions of the two mechanisms are discussed in detail in SI Appendix, and they are validated by using numerical simulations as shown in SI Appendix, Figs. S4C and S5B.

Due to the power law tail for ${\mathrm{\Gamma }}_{\mathcal{C}}(r)$ with exponent β, the mean distance to find the trail, $\overline{r}$, is determined by the upper or lower cutoff for $\beta <1$ or $\beta >1$, respectively. Since the upper cutoff is much larger than the lower cutoff, it is more convenient for the tracker to use $\beta >1$. By using the self-consistency condition $\sigma =\overline{r}/2\xi$ in the curvature-dominated regime and the relation $\overline{r}\simeq v{\omega }^{-1}\left(1+e^{-\beta }/(\beta -1)\right)$, we obtain for the maximum speed

$$v^{*}=\omega \sqrt{a\xi /{\mathrm{\Theta }}_0}{\max }_{\beta }{\left(\beta +\frac{\beta e^{-\beta }}{(\beta -1)}\right)}^{-1}\simeq 0.47\omega \sqrt{a\xi /{\mathrm{\Theta }}_0},$$ [19]

where the maximum occurs at ${\beta }^{*}=1.63$ (SI Appendix, Fig. S5C). The mean number of samples to find the trail at this optimum is $\approx 1.3$, indicating that the optimal strategy is for the agent to move slow enough that it typically finds the trail within one or two samples.

Intuition for the above results holds quite generally, including for other statistical ensembles of trails and for nonconical casting envelopes. The latter is relevant because we expect that a slowdown in the radial direction and a widening casting envelope may increase the likelihood of finding the trail. Relaxing the constraints of fixed radial speed and conical envelopes constitutes the aim of the next section.

Bellman Optimization of the Casting Strategy

Our final step is to formulate the problem of optimizing the casting strategy. The geometry of the search is shown in SI Appendix, Fig. S6A. To simplify, we parametrize the path by the set of turning points $\{r_i,{\theta }_i\}$. We assume a fixed speed v (note that the average radial speed depends on the strategy), which is set later based on the probability of losing the trail. As discussed in the text, we optimize the average tracking speed, $\langle L/T\rangle$, after constraining the IDI, $\langle L\rangle$, using a Lagrange multiplier Λ (Eq. 4). This optimization can be recast as a Bellman-type dynamic programming problem by breaking the problem up into discrete steps, each corresponding to a cast from $\{r_i,{\theta }_i\}\to \{r_{i+1},{\theta }_{i+1}\}$ (denoted $\{r\prime ,\theta \prime \}\to \{r,\theta \}$ below for conciseness):

$$\mathcal{V}(r\prime ,t\prime ,P\prime (\varphi ))={\max }_{r,\theta }\left[\left(\frac{r}{t}-\mathrm{\Lambda }r\right)(1-{\overline{\mathrm{\Gamma }}}_{r,\theta })+{\overline{\mathrm{\Gamma }}}_{r,\theta }\hspace{0.17em}\mathcal{V}\left(r,t,P_{r,\theta }(\varphi )\right)\right],$$ [20]

where the required reward function in Eq. 4 is $V_{v,\mathrm{\Lambda }}=\mathcal{V}(0,0,P_0(\varphi ))$ with the Gaussian normal prior $P_0(\varphi )=\mathcal{N}(0,{\sigma }^2)$. The first term in the square bracket returns the reward if the trail is found (weighted by the detection probability $1-{\overline{\mathrm{\Gamma }}}_{r,\theta }$), while the second term advances to the next cast if detection failed. The Bellman equation thus relates $\mathcal{V}$ before and after a cast by updating the current state variables, $r\prime ,t\prime ,P\prime (\varphi )$. The time elapsed is updated as $t\to t\prime +(r-r\prime +2r|\theta |)/v$, where we approximate $|\theta \prime -\theta |\sim 2\left|\theta \right|$ for simplicity. The prior $P\prime (\varphi )$ is updated using Bayes’ rule $P_{r,\theta }(\varphi )={\overline{\mathrm{\Gamma }}}_{r,\theta }(\varphi )P\prime (\varphi )/{\overline{\mathrm{\Gamma }}}_{r,\theta }$, where ${\overline{\mathrm{\Gamma }}}_{r,\theta }(\varphi )$ is the nondetection probability given the initial trail heading $\varphi$ and ${\overline{\mathrm{\Gamma }}}_{r,\theta }={\langle {\overline{\mathrm{\Gamma }}}_{r,\theta }(\varphi )\rangle }_{\varphi }$ is the normalization.

It remains to calculate ${\overline{\mathrm{\Gamma }}}_{r,\theta }(\varphi )$, which depends on the forward model of the trail. We consider generally that the trail’s azimuthal position expands as ${\sigma }_{\text{fwd}}(r)$ [i.e., a trail initially headed along $\varphi$ is located at the azimuthal position, $\varphi +\mathrm{\Delta }\varphi$, where $\mathrm{\Delta }\varphi \sim \mathcal{N}(0,{\sigma }_{\text{fwd}}^2(r))$ ]. For the GWLC model, we show in SI Appendix that ${\sigma }_{\text{fwd}}(r)=\sigma (r)$, where $\sigma (r)$ is given in Eq. 16. Since the time spent on the trail within the casting envelope in a single cast is a / v, we have ${\overline{\mathrm{\Gamma }}}_{r,\theta }(\varphi )=e^{-a\omega /v}$ if $|\varphi +\mathrm{\Delta }\varphi |<\theta$ and equal to unity otherwise. Since $\mathrm{\Delta }\varphi$ is normal distributed, we have ${\langle \mathrm{1}(|\varphi +\mathrm{\Delta }\varphi |<|\theta |)\rangle }_{\mathrm{\Delta }\varphi }=\mathrm{\Phi }\left(\frac{\theta -\varphi }{{\sigma }_{\text{fwd}}(r)}\right)-\mathrm{\Phi }\left(\frac{-\theta -\varphi }{{\sigma }_{\text{fwd}}(r)}\right)$, where $\mathrm{\Phi }$ is the normal cumulative distribution function. We finally approximate ${\overline{\mathrm{\Gamma }}}_{r,\theta }(\varphi )\approx e^{-\frac{a\omega }{v}{\langle \mathrm{1}(|\varphi +\mathrm{\Delta }\varphi |<|\theta |)\rangle }_{\mathrm{\Delta }\varphi }}$ and compute the expectation over $\varphi$ in ${\overline{\mathrm{\Gamma }}}_{r,\theta }$ numerically. The previous approximation greatly simplifies the optimization as $P(\varphi )$ is then a sufficient statistic for past measurements (thus allowing the decomposition in Eq. 20) and yet, captures the effect of the trail’s widening trajectory on the optimized casting strategy.

At each casting step, we optimize (using standard black box optimization methods using the SciPy library in Python) for $\mathrm{\Delta }r=r-r\prime >0$ and θ by expanding Eq. 20 two steps further into the future. The optimization then involves six variables, the immediate pair $\mathrm{\Delta }r,\theta$ and the subsequent two pairs. The optimized immediate pair is then used for updating as detailed above, and the process is repeated. Optimizing more than two steps did not yield different results as the ${\overline{\mathrm{\Gamma }}}_{r,\theta }$ factors introduce an effective planning horizon, effectively suppressing contributions from future rewards beyond two steps. For less than two steps, optimization is “greedy,” and the corresponding landscape was found to have a qualitatively different, shallow landscape around the optimum. After completing the optimization procedure, the constraint of fixed $\langle L\rangle$ is imposed by varying v and the Lagrange multiplier Λ (SI Appendix, Fig. S6B).

Data Availability

There are no data underlying this work.