Optimal control of aging in complex networks

Significance

Aging in many complex systems composed of interacting components leads to decay and eventual collapse/death. Repair delays this process at a cost, suggesting a trade-off between the cost of repair and the benefit of health and longevity. Using an interdependent network model of a complex system, we introduce a control theoretic and learning framework for maximizing longevity at minimal repair cost and determine the optimal maintenance schedule for the system. Our approach should be relevant to determining checking schedules for complex engineered and living systems.

Abstract

Many complex systems experience damage accumulation, which leads to aging, manifest as an increasing probability of system collapse with time. This naturally raises the question of how to maximize health and longevity in an aging system at minimal cost of maintenance and intervention. Here, we pose this question in the context of a simple interdependent network model of aging in complex systems and show that it exhibits cascading failures. We then use both optimal control theory and reinforcement learning alongside a combination of analysis and simulation to determine optimal maintenance protocols. These protocols may motivate the rational design of strategies for promoting longevity in aging complex systems with potential applications in therapeutic schedules and engineered system maintenance.

Aging is the process of damage accumulation with time that is responsible for an increasing susceptibility to death or decay (1). Many complex systems that consist of multiple interacting components (2) (e.g., biological organisms and artificially engineered systems) experience aging. Indeed, models of the interdependence between components of a system implemented in a network (3) show aspects of aging and eventual system-wide catastrophe and death. This is because when components are interdependent, the failure of one component may adversely affect its dependents. The dynamics of these processes have been the focus of many recent studies (4–7), exhibit temporal scaling (8, 9) and failure cascades, and reproduce empirical survivorship curves for many biological organisms and technological devices (4).

Understanding the onset of aging in network models points toward a central question in the field (10): how can one control aging in complex systems through interventions associated with repair and maintenance, with the eventual goal of designing strategies for increasing longevity? Available control strategies in networks are primarily for single nodes (11) and sets of driver nodes (12, 13), and they largely fall into three classes: network design (14, 15), edge and node removal at onset of cascade (16), and time-dependent edge weight distribution (17, 18). Complementing these approaches, in reliability engineering, there are maintenance policies for deteriorating multiunit systems (19–22) that include opportunistic repair (23) and group and block replacement (24) for systems with economic and structural dependencies between components (20, 21, 25). However, aging systems are primarily characterized by failure dependencies between components. Only very special repair policies have been optimized for failure-dependent complex systems (26), and most are restricted to systems composed of few units (20, 21), or with strong assumptions about the underlying failure distribution without consideration for the dynamics of individual network components from which they emerge (22, 27).

Here, we deploy an optimal control framework to determine strategies to delay aging in a tractable and realistic model for the failure of interdependent networks (4). A combination of numerical simulations and analysis shows how the microscopic dynamics of individual network components that are capable of stochastic failure determine how system-level macroscopic dynamics of decreasing vitality and failure cascades follow. We then introduce the notion of repair in such a network that can lead to a delay in aging, but at a cost. This can be couched in the framework of optimal control theory (28) and allows us to determine strategies to delay aging in these interdependent networks analytically in the linearized regime and computationally in the nonlinear regime. To understand the implications of our results, we then deploy a simple reinforcement learning scheme (29) to determine the parameters associated with explicit temporal repair protocols that determine the efficacy of drugs on the longevity of a classic model organism used in aging studies—the nematode Caenorhabditis elegans.

Network Model of Aging and Repair

Computational Model.

Our computational model of aging starts with the consideration of a network with $N$ nodes representing the individual components of the complex system and edges between nodes representing interdependencies between the individual components (Fig. 1A). The main network structure used in this study is the Gilbert $G\left(N,p\right)$ random graph (30); in this network, edges between any two nodes occur with probability $p$, where the mean node degree is $z=p\left(N-1\right)$. We also explore Erdős–Rényi $G\left(N,m\right)$ random networks (31) and Barabási–Albert scale-free networks (32); these structures produce qualitatively similar results as compared with the Gilbert random graph (SI Appendix, Fig. S6). In the model, each node is assigned an initial state of binary value $x_i\in \left\{\mathrm{0,1}\right\}$ with probabilities $P\left(x_i=0\right)=d$ and $P\left(x_i=1\right)=1-d$, where $d$ denotes the prenatal damage of the complex system at birth. The state of a node represents its functionality, where $x_i=1$ denotes a vital, functional $i$th node and $x_i=0$ denotes a dead, failed $i$th node.

Fig. 1.

Computational network model of aging and repair. (A) Schematic representation of the network model of aging, represented by a network, where nodes denote components and edges denote interdependencies between these components. The network aging algorithm is portrayed in a smaller subsection of the network. At each time step, nodes are failed with probability $f$, repaired with probability $r$, and failed if their fraction of vital providers is less than $I$. (B) Simulated cascading failures in a Gilbert random model ($p=0.1$, $N=\mathrm{1,000}$, $f=0.025$, $r=0$, $d=0$, $I=0.5$). Faint blue lines refer to 50 individual vitality trajectories; the solid blue line is the mean vitality $\mathrm{\Phi }\left(t\right)$; the dashed magenta line is the analytic solution to the linear model, Eq. 5, where $I=0$; and the solid gold line is the numeric solution to the nonlinear model, Eq. 4. (C) Network repair at $r=0.025$ from $T_1=10$ to $T_2=40$ (gray) delays network failure and improves mean vitality for 1,000 networks as compared with nonrepaired networks (blue). (D–F) Fluctuation of failure times. (D) Optimal repair results in increased network failure time variance (gray) as compared with no repair (dark blue). Parameters: $f=0.025$, $r=0.01$, $\alpha =10$. Also shown are the mean failure time $\mu$ and the standard deviation $\sigma$ in the absence and presence of repair. (E) Network failure time distributions for different values of $f$ are described by the Weibull distribution (solid lines) (see SI Appendix, section S3). (F) Linear regression (dotted line; slope = 0.0975) of mean $\mu$ and SD $\sigma$ of network failure times for different values of $f$.

The network is then allowed to age via a simple iterative algorithm (SI Appendix, Algorithm 1) through the following actions: 1) each node fails with probability $f$; 2) nodes are repaired with probability $r$; 3) a node fails if the fraction of vital providers (i.e., functional neighboring nodes) is less than $I$; 4) the network vitality is calculated using the expression $\varphi \left(t\right)=\frac{1}{N}{\sum }_{i=1}^N{x}_i$; and 5) the system fails if $\varphi \left(t\right)<0.1$ (Fig. 1A). Here, $I$ is a measure of the interdependence between the system components and denotes the threshold fraction of vital providers required for a node to stay alive. $I=0$ corresponds to a collection of $N$ independent components, and if the vital fraction is less than $I$, then the node automatically fails.

Our model reproduces the characteristic cascading failures that are present in the computational models of breakdown of complex systems (4, 14, 16, 33). In a representative simulation, the vitality $\varphi \left(t\right)$ of the system decreases slowly in the linear regime before collapsing rapidly after a critical vitality value, ${\varphi }_c$ (Fig. 1B). The cascading failure is observed in all three graph structures (SI Appendix, Fig. S6). This sudden decrease in system vitality is similar to the compression of morbidity that is observed during late life for humans and many other biological organisms (34). Movie S1 shows a two-dimensional visualization of network failure. Our simulations allow us to go beyond the mean-field theory and look at the probability distribution of failure events, defined in Fig. 1D in terms of the mean time for failure $\mu$ and the standard deviation in the failure time $\sigma$. In Fig. 1E, we show how these parameters vary with the failure rate $f$, and in Fig. 1F, we see that the standard deviation is linearly correlated with the mean failure time, consistent with a Weibull distribution (see SI Appendix, section S3).

Nonlinear Theory of Network Aging.

To complement our computational model of aging networks, we also construct an effective equation for the average network vitality measured over several realizations, $\mathrm{\Phi }\left(t\right)=⟨\varphi \left(t\right)⟩$. A mean-field model for the average vitality as a function of time may then be written as

$$\frac{d\mathrm{\Phi }}{dt}=-f_{\mathrm{t}\mathrm{o}\mathrm{t}}\mathrm{\Phi }+r_{\mathrm{t}\mathrm{o}\mathrm{t}}\left(1-\mathrm{\Phi }\right),$$ [1]

where $f_{\mathrm{t}\mathrm{o}\mathrm{t}}\mathrm{\Phi }$ is the total rate of node failure and $r_{\mathrm{t}\mathrm{o}\mathrm{t}}\left(1-\mathrm{\Phi }\right)$ is the total rate of repair. It is important to note that $f_{\mathrm{t}\mathrm{o}\mathrm{t}}$ and $r_{\mathrm{t}\mathrm{o}\mathrm{t}}$ denote the collective aspects of the network and are thus different from the respective intrinsic failure and repair rates $f$ and $r$ of nodes. They thus account for interdependence between nodes. To understand the relation between these variables, we note that a node fails for one of two reasons: 1) it fails with intrinsic rate $f$, or 2) it fails if the fraction of its vital providers falls below $I$ (i.e., failure cascade). At leading order in failure rate $f$, we can neglect the simultaneous failure of two or more nodes at any time point; hence, induced failure occurs in one step, when the node is left with the minimum number of vital providers, and then, one of these vital providers fails. The total rate of node failure is thus given by the sum of the intrinsic failure rate $f\mathrm{\Phi }$ and the rate of failure of the last vital provider $f_{\mathrm{t}\mathrm{o}\mathrm{t}}\mathrm{\Phi }=f\mathrm{\Phi }+k\left(1-f\right)\hspace{0.17em}{f}_{\mathrm{t}\mathrm{o}\mathrm{t}}\hspace{0.17em}m\left(I,\mathrm{\Phi }\right)\mathrm{\Phi }$, where $k=zI$ is the minimum number vital providers required for a node to function, $z=p\left(N-1\right)$ is the average number of edges between nodes (for a Gilbert random graph), and

$$m\left(I,\mathrm{\Phi }\right)=\left(\begin{array}{c}\hfill z\hfill \\ \hfill k\hfill \end{array}\right){\mathrm{\Phi }}^k{\left(1-\mathrm{\Phi }\right)}^{z-k}$$ [2]

describes the (mean-field) probability that a node is left with $k$ vital providers. We thus obtain the total rate of failure as $f_{\mathrm{t}\mathrm{o}\mathrm{t}}=\frac{f}{1-km\left(I,\mathrm{\Phi }\right)\left(1-f\right)}$. Similar arguments can be employed to determine the total rate of repair. A node can be repaired only if the following two conditions are met: 1) the node is failed, and 2) the node is connected to at least the minimum fraction $I$ of vital providers required for it to function after repaired. The total rate of repair is thus the product of the intrinsic rate of repair, $r\left(1-\mathrm{\Phi }\right)$, and the probability $h\left(I,\mathrm{\Phi }\right)$ that the node is connected to at least $k=zI$ vital providers:

$$h\left(I,\mathrm{\Phi }\right)=\sum _{j=k}^z\left(\begin{array}{c}\hfill z\hfill \\ \hfill j\hfill \end{array}\right){\mathrm{\Phi }}^j{\left(1-\mathrm{\Phi }\right)}^{z-j}.$$ [3]

In summary, we arrive at

$$\frac{d\mathrm{\Phi }}{dt}=-\frac{f\mathrm{\Phi }}{1-km\left(I,\mathrm{\Phi }\right)\left(1-f\right)}+r\hspace{0.17em}h\left(I,\mathrm{\Phi }\right)\left(1-\mathrm{\Phi }\right),$$ [4]

where $f$ and $r$ are the intrinsic frequencies of failure and repair, respectively, and interdependence between nodes is captured in this mean-field equation by the nonlinear functions $m\left(I,\mathrm{\Phi }\right)$ and $h\left(I,\mathrm{\Phi }\right)$. In SI Appendix, Fig. S1, we compare the mean-field model Eq. 4 and the network simulations (Fig. 1B). Analytically, we see that the solution to Eq. 4 describes an average vitality that decreases slowly at early times. In the limit when the system is away from collapse ($ft\ll 1$), Eq. 4 can be linearized and approximated to leading order as

$$\frac{d\mathrm{\Phi }}{dt}=-f\mathrm{\Phi }+r\left(1-\mathrm{\Phi }\right).$$ [5]

This leads to an exponentially decaying vitality (Fig. 1B). At later times, the average vitality exhibits failure cascade and rapid collapse after a critical vitality value ${\mathrm{\Phi }}_c$ is reached (Fig. 1B). This effect originates when the denominator in the first term on the right-hand side of Eq. 4 becomes small, which causes the effective failure rate to blow up; thus, an estimate for the critical fraction for failure cascade can be obtained by maximizing $m\left(I,\mathrm{\Phi }\right)$ over $\mathrm{\Phi }$, which yields ${\mathrm{\Phi }}_c=I$ (SI Appendix, section S1).

Optimal Control of Network Aging

Having a qualitative understanding of the forward problem of how aging arises in interdependent networks, we now turn to the problem of controlling the progressive aging of a network by varying the repair rate, subject to some constraints.

Optimal Repair Protocols.

For an interdependent network that ages according to Eq. 4, our goal is to design optimal repair protocols [i.e., replace the constant repair frequency $r$ in Eq. 4 by a time-dependent unknown repair rate $r\left(t\right)$ to regulate network vitality]. Since high vitality is expected to correspond to a “benefit,” while repair actions come with a “cost,” we introduce the following cost function to capture this balance between network vitality and repair:

$$\text{Cost}={\int }_0^T{e}^{-\gamma t}\hspace{0.17em}\mathcal{C}\left(\mathrm{\Phi }\left(t\right),r\left(t\right)\right)dt,$$ [6]

where $T$ is the final time and $\mathcal{C}$ is a monotonically decreasing function of vitality $\mathrm{\Phi }\left(t\right)$ and a monotonically increasing function of repair $r\left(t\right)$. The exponential term describes the situation when future values of the cost are discounted, where $\gamma \ge 0$ is the discount rate. To balance the cost of repair and the benefit of vitality, we focus here on a simple linear cost function $\mathcal{C}=\alpha \hspace{0.17em}r\left(t\right)-\mathrm{\Phi }\left(t\right)$, where $\alpha$ is the relative cost of repair. The first term describes the total cost for repair as the integral of the repair protocol in time, while the second term is the gain from vitality; the constant $\alpha$ describes the relative importance of the two terms in the cost function. The goal of the optimal control problem defined by Eqs. 4 and 6 is to find the repair protocol $r\left(t\right)$ that minimizes the cost function Eq. 6 while satisfying the evolution equation for vitality Eq. 4 (SI Appendix, section S8 has a discussion of alternative cost functions and the effect of nonlinear cost functionals).

We solve this optimal control problem for a network with initial vitality $\mathrm{\Phi }\left(t=0\right)=1-d$ using the framework of optimal control theory and Pontryagin’s principle (28) (SI Appendix, section S2 has details). Since the optimal control problem is linear in the repair rate $r\left(t\right)$, the optimal repair protocol will correspond to a bang-bang control that switches between $r\left(t\right)=0$ (no repair) and $r\left(t\right)=r$ (maximal repair). Repair is turned on when

$$h\left(I,\mathrm{\Phi }\right)\left(1-\mathrm{\Phi }\right)>\frac{\alpha }{\left|\lambda \right|},$$ [7]

where $\lambda$ is a time-dependent costate variable, which is determined as the solution to SI Appendix, Eq. S23 (SI Appendix, section S2.2 has a derivation, and SI Appendix, section S2.4 has a discussion on singular arcs). Eq. 7 states that the optimal decision to repair depends on two parameters: 1) the repairable fraction of nodes, $h\left(I,\mathrm{\Phi }\right)\left(1-\mathrm{\Phi }\right)$, and 2) a time-dependent threshold $\alpha /\left|\lambda \right|$, which depends on the relative cost of repair $\alpha$. The repairable fraction increases with time as nodes in the network fail and/or become increasingly susceptible to failure cascades; on the other hand, the threshold for the repairable fraction also increases with time as the system ages, leading to a smaller window of repair.

Linear Control Theory.

To gain an understanding of how the optimal repair protocol depends on the physical parameters, we focus first on the linearized limit, Eq. 5, valid away from vitality collapse. In fact, explicit analytical expressions for optimal protocols can be obtained in this case. Condition Eq. 7 leads to nonmonotonic optimal repair protocols characterized by a waiting time for repair, followed by an intermediate period where repair is preferable and a terminal phase where the repair rate is set again to zero (SI Appendix, Fig. S2):

$$r\left(t\right)=\left\{\begin{array}{cc}0,\hspace{1em}\hfill & t<T_1\hspace{0.17em}\text{and}\hspace{0.17em}t>T_2\hfill \\ r,\hspace{1em}\hfill & T_1<t<T_2\hfill \end{array}\right.,$$ [8]

where $T_1$ and $T_2$ are switching times, given by (SI Appendix, section S2 has a derivation)

$$T_1\simeq \frac{1}{f}\log \left[\frac{1-d}{1-\alpha \left(f+r+\gamma \right)}\right]$$ [9a]

$$T_2\simeq T-\frac{1}{f+\gamma }\log \left[\frac{1}{1-\alpha \left(f+r\right)\left(f+\gamma \right)/f}\right].$$ [9b]

The dependence of $T_1$ and $T_2$ on the failure rate $f$, repair rate $r$, and cost of repair $\alpha$ is shown in Fig. 2. The optimal repair protocol in time consists of an initial phase when system vitality is high and no repair is necessary and a repair period that is initiated at time $T_1$ and persists until time $T_2$. For $\gamma =0$ and $d=0$ (corresponding to a healthy organism), the repair protocol is symmetric with respect to the end time $T$ since $T_1=T-T_2$. The protocol is no longer symmetric with respect to $T$ when $d>0$; in particular, while the initial vitality level does not affect the end time $T_2$, the start time $T_1$ decreases with increasing $d$, implying that the optimal repair protocol starts earlier and lasts for longer as the initial vitality of the system decreases. There is a critical value for initial vitality, $\mathrm{\Phi }\left(t=0\right)<1-d_c=1-\alpha \left(f+r+\gamma \right)$, below which the optimal repair protocol starts right away. Robust achievability of optimal protocols depends on the curvature of the cost function around the optimum Eq. 9, which for $\gamma =0$ is given by $\simeq fr\left[1-\alpha \left(f+r\right)\right]/\left(f+r\right)$ (SI Appendix, section S2.3.1). We also characterized longevity gain resulting from optimal protocols as a function of $\alpha$ and $\gamma$ (SI Appendix, Fig. S3).

Fig. 2.

Optimal repair protocols to maximize health span at minimum intervention cost. (A) Schematic representation of optimal bang-bang repair protocol $r\left(t\right)$ with repair start time $T_1$ and repair stop time $T_2$ as showcased in Eq. 8 for the linear regime. (B) The repair duration (shaded blue) is dependent on the failure rate $f$ and disappears for small $f$ and large $f$ as calculated from Eq. 9. (C) The repair duration monotonically decreases with increased maximum repair rate $r$. (D) The repair duration decreases with increased cost of repair $\alpha$ and disappears for large $\alpha$. The default parameters used for A–D were $N=1000$, $p=0.1$, $f=0.025$, $r=0.01$, $\alpha =10$, $\gamma =0$, $T=100$, $d=0$, $I=0$. (E) Optimal repair protocol for an interdependent network. Solid lines correspond to the numerical solution to the optimal control problem. Scatter points correspond to the optimal switching times obtained from a grid search on the computational model. The default parameters used were $N=1000$, $p=0.1$, $f=0.025$, $r=0.01$, $\gamma =0$, $T=100$, $d=0$.

In the infinite horizon limit $T\to \infty$ and $\gamma >0$, we enter a regime where the optimal solution for repair maximizes the discounted health of the system over an indefinite period under a cost constraint. Biologically, this is equivalent to optimizing longevity as compared with health span for finite $T$, while considering a discount factor resulting from extrinsic mortality (35). Since $T_2\to \infty$, the infinite horizon repair protocol is characterized by a single switching time $T_1$, after which the system is repaired in perpetuity.

Thus far, we have focused on the simple linear cost function. Exploring nonlinear cost functions leads to optimal repair protocols that are no longer of bang-bang type but are still nonmonotonic in time (SI Appendix, section S8), with initial and terminal phases of low repair and an intermediate region of higher repair (SI Appendix, Fig. S8). Additional extensions may be motivated by future experiments and might involve considering a terminal cost for vitality, including nonlinearities in vitality and/or repair rate (SI Appendix, section S8), or introducing additional variables, such as node checking and associated cost (SI Appendix, section S9).

Phase Diagram for Repair.

A question of some interest is the determination of the conditions under which a repair protocol is advisable. From Eq. 9, it follows that since $T_1$ must, by definition, be smaller than $T_2$, a repair protocol exists for $d=0$ and $\gamma =0$ only if

$$fT\ge 2\log \left[\frac{1}{1-\alpha \left(f+r\right)}\right].$$ [10]

Eq. 10 results in a phase diagram separating a region of “repair” from a region of “no repair,” where repair is too costly, as a function of two relevant dimensionless parameters $\alpha \left(f+r\right)$ and $fT$. As a function of failure frequency $f$ and at constant values of $\alpha ,r$, and $T$, Eq. 10 predicts the existence of regions of low ($fT\ll 1$) and high failure rates ($fT\gg 1$), respectively, where the best option is no repair (Fig. 2B). This behavior follows intuition; when failure rate is low, vitality remains high over the interval $\left[0,T\right]$, such that the cost of repair would be unnecessarily large compared with the benefit associated with increased vitality. Similarly, when the failure rate is large, a significant improvement of vitality would require an insurmountable cost of repair. As the repair rate $r$ increases, Eq. 10 predicts a rapidly shrinking window of repair due to the combined effect of increasing the effectiveness of and associated cost ($\alpha r$) of repair (Fig. 2C). As the cost of repair $\alpha$ increases, Eq. 10 similarly predicts a decreasing window of repair (Fig. 2D) that results from an increasing cost burden. There exists a critical value for the repair cost, ${\alpha }_c=1/\left(f+r\right)$, above which there is no repair.

Interdependent Networks.

For networks with interdependent components, the optimal protocols are still bang bang, and the switching times can be calculated using Eq. 7. Notably, increasing the interdependence ($I\ge 0$) between components provided qualitatively similar strategies for maintaining optimal health span (finite $T$) as the linear theory. Our theory predicts that the window of repair increases with interdependence in order to compensate for the accelerated aging and reduced response to repair in interdependent networks. Increasing $I$ has little effect on the switching time $T_1$ since at high vitality, the interdependent system is close to the linear theory. However, as $I$ increases, the repairable fraction $h\left(\mathrm{\Phi },I\right)$ and the effective repair rate decrease monotonically with $I$ for fixed $\mathrm{\Phi }$, which results in an increasing repair stop time $T_2$. We ran computational simulations of the network model to validate the predicted optimal repair policies as interdependence is increased (SI Appendix, section S4). The results shown in Fig. 2E agree with the optimal policies calculated using Eq. 7 (solid lines).

Fluctuations in Failure Times.

Using our framework, we have measured the extent of fluctuations in failure times with and without repair (Fig. 1D). The resulting distribution of failure times is described by the Weibull distribution (SI Appendix, section S3). Interestingly, we find that, in addition to increasing the average failure time, application of the finite horizon optimal repair protocol results in a broader distribution of network failure times (Fig. 1 C–E). This effect emerges naturally from the Weibull distribution, in which the mean failure time is linearly correlated with the SD in failure times for different values of $f$ (Fig. 1F).

Role of Network Topology.

We have also studied optimal protocols numerically for Erdős–Rényi $G\left(N,m\right)$ random networks (31) and Barabási–Albert scale-free networks (32). The aging dynamics are highly similar between the three network models investigated (SI Appendix, Fig. S6 A–C). For all random and scale-free networks, we observe no significant qualitative differences in the optimal repair protocols (SI Appendix, Fig. S6 D and E), indicating that our protocols are robust and may be applicable to a diverse range of complex systems.

A Generalization and an Application

Our computational and theoretical model of aging and its control in a complex network naturally raises the question of whether the optimal control policy that we determined can be iteratively learned and applied to a real system.

Reinforcement Learning Approach to Interdependent Network Aging Control.

Optimal control strategies rely on knowledge of the model and a cost function, both of which are hard to crystallize into quantitative form in many biological systems. An alternative strategy is to ask whether the system is able to learn the optimal repair protocol for aging via an iterative procedure. This is tantamount to direct adaptive optimal control (36), embodied in reinforcement learning, a process by which a system is able to optimize its actions by interacting with its environment. Optimization occurs iteratively on a trial and error basis by reinforcing actions that maximize reward and/or minimize punishment. We use a relatively simple version of this algorithm known as Q learning (Fig. 3A and SI Appendix, Fig. S5) (29), which is a model-free alternative to dynamic programming models of the Bellman equation (37). This method consists of creating a Q matrix, $Q=\left\{\varphi ,r\left(\varphi \right)\right\}$, which serves as a look-up table of vitality states $\varphi$ and values associated with each possible action, $r\left(\varphi \right)=0$ or $r\left(\varphi \right)=r$. In each training episode, a healthy ($d=0$) network is initialized. At each time step, the network is subjected to the aging algorithm, and the agent exploits network repair for the greatest-valued choice of repair at the given vitality of the system with probability $1-e^{-{\lambda }_{\text{exp}}q}$ where $q$ is the number of episodes elapsed. The agent explores with probability $e^{-{\lambda }_{\text{exp}}q}$. A reward $R$ is calculated and used to update the state-action value in the Q matrix according to the rule (29)

$$\begin{array}{ll}\hfill Q\left({\varphi }_t,r_t\right)& \leftarrow Q\left({\varphi }_t,r_t\right)+\beta [R_t+{\gamma }_Q\underset{\rho }{max}\left\{Q\left({\varphi }_{t+1},\rho \right)\right\}\hfill \\ \hfill & \hspace{1em}\hspace{0.17em}-Q\left({\varphi }_t,r_t\right)],\hfill \end{array}$$

$$R_t={\varphi }_t-\alpha r_t,$$

where $\alpha$ is the cost of repair, $\beta$ is the learning rate, and ${\gamma }_Q$ is the Q-learning discount factor that is related to the optimal control through $\gamma =-\log {\gamma }_Q$. The learning rate exponentially decays as $\beta =e^{-{\lambda }_{\beta }q}$. An episode ends when the network fails (i.e., $\varphi <0.1$). The Q-learning model iterates through learning episodes until qualitative convergence of the Q matrix is achieved. The optimal protocol is defined as the maximal Q-valued trajectory traveled by a network through $\left(\varphi ,r\right)$ space.

Fig. 3.

Optimal repair protocols using reinforcement learning. (A) High-level schematic of reinforcement learning algorithm for optimal control of network aging. SI Appendix, section S5 has further Q-learning model details. (B) Optimal $T_1$ as a function of the cost of repair $\alpha$ for the reinforcement learning (gray circles; error bars span 75% CI, $N=50$ realizations) and the theoretical solution (dotted magenta line) (Eq. 9). Models used $N=1000$, $p=0.1$, $f=0.025$, $r=0.01$, $\gamma =0.975$, $I=0$. (Inset) The learned repair protocol (represented as points) is bang bang, matches closely with the theoretically optimal repair protocol (line) (Eq. 8), and is characterized by a single repair switching time $T_1$. Parameters used were $f=0.0367$, $r=0.01$, $\alpha =10$, $\gamma =0.975$, $I=0$, $d=0$. (C) Switching time (in days) determined for $\alpha$-ketoglutarate treatment of C. elegans (38) using estimated $f=0.004$, $r=0.043$, $I=0.8$, and different choices of $\alpha$. Q learning was used on network models ($N=200$, $p=0.1$). Error bars span 75% CI.

Using this method, the Q-learned repair policy converges to optimal repair protocols that are bang bang (Fig. 3B, Inset) and closely match the predicted switching time $T_1$ from the analytic theory for different values of $\alpha$ (Fig. 3B). These results suggest that the optimal protocols for repair can be obtained through simple iterative learning and highlight the potential of Q learning as a method to approximate optimal repair protocols for complicated systems in which no analytic description of the aging dynamics is available.

Quantifying the Protocols for Extending Longevity of C. elegans.

We now deploy this approach using data from a model organism in aging research, the nematode C. elegans. We used C. elegans life span data (38–41) in the presence of a metabolite $\alpha$-ketoglutarate that significantly extend C. elegans life span from which we estimate the parameters in our minimal model corresponding to the failure rate $f$, interdependency $I$, and $\alpha$-ketoglutarate repair rate $r$ (SI Appendix, section S7). Using Q learning with these fitted parameters, we predict optimal protocols for this system as a function of the cost of repair $\alpha$ (Fig. 3C). Consistent with our model, the switching time for $\alpha$-ketoglutarate treatment increases with the cost of repair $\alpha$. These results suggest that life-extending treatments that work through repair or replacement may be determined through measurable parameters. However, understanding the dependence of these strategies on the choice of the cost remains an outstanding question.

Discussion

Although aging in real biological and technological systems is a consequence of complex biochemical and mechanical processes, here we have abstracted a minimal model designed to capture the essential ingredients that give rise to aging in a complex system—modular units (nodes) that are linked to each other via a set of edges modeled as an interdependent network subject to nodal failure and repair. Our model shows the emergence of failure cascades, a hallmark of such systems, which we can understand using an analytic approach that accounts for both the observed mean and extreme value statistics of failure. We then deployed a simple scheme for repair in such interdependent systems, couched as an optimal control problem to slow down aging. We first used a model-dependent strategy to determine explicit optimal repair protocols for aging interdependent systems characterized by a failure rate $f$, repair rate $r$, and interdependency $I$. We also demonstrated that a model-free approach using reinforcement learning converges to these optimal repair protocols and can therefore be leveraged to approximate optimal repair strategies in an iterative manner. This allowed us to estimate model parameters for the efficacy of a drug used to increase the longevity of a model organism in aging studies—the nematode C. elegans.

We conclude with some implications of our work to the biological problem of determining optimal protocols for life-extending treatments such as the clearance of senescent cells. Senescent cells enter a permanent, nondividing state and adopt an altered secretory profile, which has been implicated in inflammation, tumorigenesis, and aging (10, 42). The presence of these cells has been shown to promote senescence in surrounding tissue (43), similar to how node failure in a network can spread due to interdependence. In contrast, the selective clearance of senescent cells (i.e., via the use of senolytic cocktails) improves physical function and survival (43, 44), without reducing either the total cell count (in human tissue) or body weight (in mouse models) (43). This suggests a rapid replacement of cleared senescent cells by healthy dividing cells, so that the application of senolytic treatments becomes analogous to node repair in an aging network. Like many life-extending compounds, senolytic cocktails may include toxicity (45, 46), which can be modeled in the cost function. The relative cost of repair $\alpha$ can be determined by separately measuring the loss of viability caused by senescence and the toxicity that results from treatment on an ensemble of healthy cells. This would allow for the deployment of our optimal repair protocols and design treatment schedules for senescent cell inhibitors and other therapeutics that target general aging processes.

Materials and Methods

There are no data associated with this paper.