Actor critic with experience replay‐based automatic treatment planning for prostate cancer intensity modulated radiotherapy

Abstract

Background

Achieving highly efficient treatment planning in intensity‐modulated radiotherapy (IMRT) is challenging due to the complex interactions between radiation beams and the human body. The introduction of artificial intelligence (AI) has automated treatment planning, significantly improving efficiency. However, existing automatic treatment planning agents often rely on supervised or unsupervised AI models that require large datasets of high‐quality patient data for training. Additionally, these networks are generally not universally applicable across patient cases from different institutions and can be vulnerable to adversarial attacks. Deep reinforcement learning (DRL), which mimics the trial‐and‐error process used by human planners, offers a promising new approach to address these challenges. 

Purpose

This work aims to develop a stochastic policy‐based DRL agent for automatic treatment planning that facilitates effective training with limited datasets, universal applicability across diverse patient datasets, and robust performance under adversarial attacks.

Methods

We employ an actor–critic with experience replay (ACER) architecture to develop the automatic treatment planning agent. This agent operates the treatment planning system (TPS) for inverse treatment planning by automatically tuning treatment planning parameters (TPPs). We use prostate cancer IMRT patient cases as our testbed, which includes one target and two organs at risk (OARs), along with 18 discrete TPP tuning actions. The network takes dose–volume histograms (DVHs) as input and outputs a policy for effective TPP tuning, accompanied by an evaluation function for that policy. Training utilizes DVHs from treatment plans generated by an in‐house TPS under randomized TPPs for a single patient case, with validation conducted on two other independent cases. Both online asynchronous learning and offline, sample‐efficient experience replay methods are employed to update the network parameters. After training, six groups, comprising more than 300 initial treatment plans drawn from three datasets, were used for testing. These groups have beam and anatomical configurations distinct from those of the training case. The ProKnow scoring system for prostate cancer IMRT, with a maximum score of 9, is used to evaluate plan quality. The robustness of the network is further assessed through adversarial attacks using the fast gradient sign method (FGSM).

Results

Despite being trained on treatment plans from a single patient case, the network converges efficiently when validated on two independent cases. For testing performance, the mean $\pm$ standard deviation of the plan scores across all test cases before ACER‐based treatment planning is $6.17\pm 1.90$. After implementing ACER‐based treatment planning, $92.29\%$ of the cases achieve a perfect score of 9, with only $6.65\%$ scoring between 8 and 9, and no cases being below 7. The corresponding mean $\pm$ standard deviation is $8.92\pm 0.29$. This performance highlights the ACER agent's high generality across patient data from various sources. Further analysis indicates that the ACER agent effectively prioritizes leading reasonable TPP tuning actions over obviously unsuitable ones by several orders of magnitude, showing its efficacy. Additionally, results from FGSM attacks demonstrate that the ACER‐based agent remains comparatively robust against various levels of perturbation.

Conclusions

We successfully trained a DRL agent using the ACER technique for high‐quality treatment planning in prostate cancer IMRT. It achieves high generality across diverse patient datasets and exhibits high robustness against adversarial attacks.

1. INTRODUCTION

Real‐time treatment planning represents a challenging problem in intensity modulated radiotherapy (IMRT) for cancer treatment. The difficulty mainly arises from the complex energy deposition properties of the radiation beam within the patient body, which often requires a careful balance between target dose coverage and normal tissue sparing. This balance can be achieved via inverse treatment planning optimization, which involves setting dose constraints for each organ and target, along with weighting factors to prioritize conflicting dose requirements. However, the optimal value set of these treatment planning parameters (TPPs) to achieve clinical acceptable plan are often initialization condition dependent, such as the patient geometries, fluence map, TPP initialization and so forth. ¹ In state‐of‐the‐art radiotherapy clinics, it requires repeatedly adjusting the TPPs by human planners to refine the inverse treatment planning optimization, which is time‐consuming and poses challenges for real‐time planning. 

In the era of artificial intelligence (AI), significant efforts have been made to address challenges in treatment planning. One approach aims to shorten the trial‐and‐error process by generating a relatively “good” initial plan using machine learning, allowing human planners to refine and optimize the plan more efficiently. ² For instance, Li et al. ² successfully employed architectures like ResNet ³ and DenseNet ⁴ to create high‐quality starting plans, thereby accelerating the overall treatment planning process. Another approach bypasses the trial‐and‐error process entirely by using machine learning to directly predict desired 3D dose distributions and corresponding 2D fluence map intensities based on patient image and contour data. ⁵ , ⁶ For example, Vandewinckele et al. ⁵ utilized a U‐Net‐based convolutional neural network (CNN) ⁷ to predict 3D dose distributions from CT scans and the contours of targets and organs. They subsequently applied another U‐Net‐based CNN to predict the 2D fluence map from the 3D dose, with or without patient image and contour data, for lung cancer IMRT cases.

Although these AI techniques have significantly reduced the time required for treatment planning, they often depend on large databases of high‐quality patient data for training due to their supervised or unsupervised learning nature. Additionally, the trained networks are frequently not universally applicable across different institutions due to data heterogeneity, ⁸ creating barriers to their widespread adoption. Furthermore, these developments typically lack adversarial attack testing, despite vulnerabilities being identified in various supervised learning algorithms used in medical applications. ⁹

Reinforcement learning (RL), ¹⁰ which mimics the trial‐and‐error learning process used by humans to achieve their goals, offers a new angle to accelerate treatment planning. Unlike other methods, the trial‐and‐error process in RL generates substantial new data samples that the algorithm can learn from, thereby reducing the dependence on large initial datasets. Recently, the application of deep neural network‐based Q‐learning, specifically deep Q‐networks (DQN), ¹¹ has shown promising results in automating treatment planning. ¹² , ¹³ , ¹⁴ , ¹⁵ , ¹⁶ , ¹⁷ For example, Shen et al. ¹² utilized DQN to develop a network that observed dose–volume histograms (DVHs) and output actions to adjust organ weighting factors in inverse treatment planning for high‐dose‐rate brachytherapy in cervical cancer. The same research group later demonstrated the feasibility of this approach for automatic tuning of TPPs in external beam IMRT for prostate cancer. ¹³ Additionally, Sprouts et al. ¹⁶ extended the DQN‐based virtual treatment planner (VTP) to adjust TPPs compatible with commercial TPS systems, achieving effective treatment planning for prostate IMRT.

Despite these promising advancements, the DQN network has inherent limitations that complicate its application to complex, clinically relevant treatment planning scenarios. In clinical settings, human planners often adjust numerous TPPs for different targets and organs at risk (OARs), resulting in a vast state‐action space that the RL algorithm must explore. In such cases, finding the optimal action concerning the Q function can be costly due to challenges like overestimation of the Q value function and difficulties in balancing exploration and exploitation. ¹⁸ To partially address these issues, Shen et al. introduced a knowledge‐guided network training strategy ¹⁴ and a hierarchical approach ¹⁵ within the DQN framework, demonstrating some success in prostate stereotactic body radiation therapy (SBRT) automatic treatment planning. ¹⁷ However, not all challenges were resolved. Additionally, continuous TPP tuning leads to a continuous action space, where DQN's effectiveness diminishes. Like other deep neural networks, DQNs are also vulnerable to adversarial attacks, ¹⁹ raising concerns about their robustness in clinical applications.

To tackle these challenges, we propose a new RL approach, actor–critic with experience replay (ACER), ²⁰ to automate the treatment planning process. The concept underlying this approach was briefly introduced in our previous conference presentation. ²¹ ACER builds on the advanced actor–critic algorithm (A3C), ²² enhancing data sampling efficiency through experience replay. ²³ In this framework, the actor functions as the policy network, aiming to maximize returns, while the critic assesses the quality of the actor's decisions. This setup inherently facilitates exploration and exploitation, and the policy gradient method allows effective exploration of both discrete and continuous action spaces. Previous studies indicate that A3C can be more resistant to adversarial attacks than DQN. ¹⁹ Based on these observations, we propose applying ACER to develop a training‐efficient, robust, and scalable agent for automatic treatment planning applications.

In the following sections, we will detail our implementation of the ACER algorithm for automating the TPP tuning process in inverse treatment planning, using prostate IMRT as a test case. Followed by it is the evaluation of its performance across different datasets, including its vulnerability to adversarial attacks.

2. METHODS AND MATERIALS

2.1. Overall architecture

The overall architecture of the ACER‐based automatic treatment planning system is similar to that of the DQN‐based system, ¹⁶ as shown in Figure 1. The process begins with the random initialization of the TPPs, which are then input into the TPS system for inverse treatment planning. The quality of the resulting treatment plan is evaluated, and if it does not meet the desired standards, both the plan and the TPPs are fed into the ACER‐based VTP system for TPP tuning. With the updated TPPs, the TPS performs inverse treatment planning optimization again. This iterative process continues until the plan quality meets the required standards or the maximum number of TPP tuning iterations is reached.

FIGURE 1.

The workflow of the ACER‐based automatic treatment planning process. Here, “TPP” represents treatment planning parameter. ACER, actor–critic with experience replay; TPP, treatment planning parameter.

In the following subsections, we will give the design details for the in‐house TPS, the ACER‐based VTP system, the testbed, the plan evaluation system, and the system performance test.

2.2. In‐house treatment planning system (TPS)

We developed an in‐house dose‐volume constraint TPS ¹⁶ following the documentation of Varian's Eclipse. ²⁴ For a treatment planning containing a single target and $N$ OARs, the objective function for the fluence map optimization can be formulated as:

$$\begin{array}{ccc}& & min\frac{1}{2}||Mx-d_p{||}_{-}^2+\frac{\lambda }{2}||{(Mx-t{d}_p)}_{V_{\mathrm{PTV}}}{||}_{+}^2\hfill \\ & & +\sum _i^N\frac{{\lambda }_i}{2}||{(M_{i}x-t_i{d}_p)}_{V_i}{||}_{+}^2,\hfill \\ & & \mathrm{s}.\mathrm{t}.x\ge 0,D_{95\%}(Mx)=d_p.\hfill \end{array}$$ (1)

In this equation, ${|·|}_{-}^2$ and ${|·|}_{+}^2$ represent the standard $l_2$ norms, which compute only the negative and positive elements, corresponding to under‐dose and over‐dose constraints, respectively. The under‐dose constraint is further reinforced that at least $95\%$ of the PTV volume receives the prescription dose. The relative importance of the respective terms is adjusted by the weighting factors $\lambda$ and ${\lambda }_i$. Regarding the other variables in the equation, $M$ represents the dose deposition matrix, $x$ is the beamlet vector, and $d_p$ is the prescription dose. $t{d}_p$ and $V_{\mathrm{PTV}}$ are upper threshold dose and the percentage of the PTV volume considered for over‐dose constraints, respectively. Similarly, $M_i$, $V_i$, and $t_i$ are the corresponding variables used for the overdose constraints on the $i$th OAR. Voxels included in $V_{\mathrm{PTV}}$ and $V_i$ always receive a higher dose than those not selected.

In summary, the free TPPs to be tuned in this in‐house TPS are $\lambda$, $V_{\mathrm{PTV}}$, $t$, ${\lambda }_i$, $V_i$, and $t_i^d$ (where $i=1,2,\text{…},N$), totaling $3(N+1)$ parameters. Given a set of TPPs, the optimization problem can be solved using the alternating direction method of multipliers (ADMM).

2.3. Actor critic with experience replay (ACER)‐based virtual treatment planner (VTP) system

2.3.1. Working principle of ACER

ACER is a type of deep reinforcement learning that integrates a deep neural network with actor–critic learning while leveraging the experience replay. ²⁰ This approach has shown superior performance in challenging environments, including the Atari57 game collection. ²⁰

In actor–critic learning, an actor agent generates decisions, while a critic agent evaluates those decisions in the context of a sequential decision‐making problem. This process involves a dynamic environment represented by a series of states $s\in \mathcal{S}$ associated with a series of possible actions $a\in \mathcal{A}$. After taking an action $a$ in state $s$, the state transitions into the next state $s^{\prime }$ with a probability $Pr\{s^{\prime }|s,a\}$, yielding a stepwise reward $r\in \mathcal{R}$. The optimal decision made by the actor agent, or the optimal policy $\pi$, can be defined as choosing those series of actions $a$ to maximize the accumulated reward for states $s$ over time $t\in (0,1,2\text{…})$. The corresponding objective function is: ²⁵

$$J(\pi )=\sum _s{\lim }_{t\to \infty }Pr\{s_t=s|s_0,\pi \}\sum _a\pi (a|s){\mathcal{R}}_s^a.$$ (2)

Representing the policy $\pi$ by a network that has parameters $\theta$, the policy parameters $\theta$ can be optimized using the policy gradient approach governed by the policy gradient theorem ²⁵ as:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill g& ={\nabla }_{\theta }J({\pi }_{\theta })\hfill \\ & =\sum _s{\lim }_{t\to \infty }Pr\{s_t=s|s_0,\pi \}\sum _a{\nabla }_{\theta }{\pi }_{\theta }(a|s)Q^{{\pi }_{\theta }}(a,s)\hfill \\ & ={\mathbb{E}}_{\theta }[Q^{{\pi }_{\theta }}(a,s){\nabla }_{\theta }\log {\pi }_{\theta }(a|s)].\hfill \end{array}\end{array}$$ (3)

Here, $Q^{\pi }(a,s)$ is the state‐action value function, serving as an effective evaluation of the policy performance.

In actor–critic learning, $Q^{\pi }(a,s)$ is estimated by the critic agent. When selecting an action $a$ in a state $s$ at step $t$, the critic estimates $Q^{\pi }(a_t,s_t)$ as the expected cumulative reward

$${\mathbb{E}}_{s_{t+1}:\infty ,a_{t+1}:\infty }\left(\sum _{i\ge 0}{\gamma }^i{r}_{t+i}|a_t=a,s_t=s\right)$$ (4)

following policy $\pi$. Here, $\gamma \in [0,1)$ is the discount factor for future rewards.

In ACER, the actor and critic agents are integrated into a deep neural network with “two heads.” One head gives the policy ${\pi }_{\theta }(a_t,s_t)$ with network parameters $\theta$, while the other outputs the estimate $Q_{{\theta }_c}(a_t,s_t)$ with network parameters ${\theta }_c$. Particularly, ACER designs the policy network to contain two parts, the distribution $f$, and the statistics ${\varphi }_{\theta }(s)$ of $f$, thus the policy can be fully represented as $\pi (·|s)=f(·|{\varphi }_{\theta }(s))$.

To solve the intrinsic instability that arises from combining online RL with deep neural network, ²² ACER employs a hybrid online and offline training strategy to optimize the network performance. Specifically, it adopts the online asynchronous updating of $\theta$ and ${\theta }_c$ by launching parallel network learners that interact with different instances of the environment. ²² This technique de‐correlates the agents' data, thereby stabilizing the learning process. Additionally, it implements the offline experience replay strategy, ²³ allowing the agent to learn from memory of experiences. Trajectories can be retrieved from the memory and weighted by importance sampling, promoting both learning stability and data efficiency.

Particular to the training of the critic network, ACER constructs the target $Q$ with retrace estimator ²⁶ as

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill Q^{\mathrm{tar}}(a_t,s_t)& =Q^{\mathrm{ret}}(a_t,s_t)\hfill \\ & =r_t+\gamma {\overline{\rho }}_{t+1}[Q^{\mathrm{ret}}(a_{t+1},s_{t+1})\hfill \\ & -Q_{{\theta }_c}(a_{t+1},s_{t+1})]+\gamma V_{{\theta }_c}(s_{t+1}).\hfill \end{array}\end{array}$$ (5)

Here, $\overline{\rho }=min(c,\rho )$ is the truncated importance weight used in experience replay, where $c$ is a constant and $\rho$ is the importance weight. During online updating, $\overline{\rho }=1$. The value function $V_{{\theta }_c}(s)$ is derived from the critic's $Q$ estimator as $V_{{\theta }_c}(s)={\mathrm{\Sigma }}_a{Q}_{{\theta }_c}(a|s)f(a|\varphi (s))$. $Q^{\mathrm{ret}}$ has been shown to have low variance and to converge effectively. With it, the critic network parameter ${\theta }_c$ is then updated as $d{\theta }_c=d{\theta }_c+{\nabla }_{{\theta }_c}{(Q^{\mathrm{ret}}(a,s)-Q_{{\theta }_c}(a,s))}^2$.

The low‐variance $Q^{\mathrm{ret}}$ is also employed to stabilize online policy updates by replacing $Q^{{\pi }_{\theta }}$ term as $Q^{\mathrm{ret}}-V$ in Equation (3). ACER further incorporates truncated importance sampling in experience replay with bias correction to enhance data efficiency while avoiding excessive bias in policy updates, which yields the offline policy gradient $g$ with respect to $\varphi$ as follows:

$$\begin{array}{ccc}\hfill g_t^{\mathrm{ACER}}& =& {\overline{\rho }}_t{\nabla }_{{\varphi }_{\theta }(s_t)}\log f(a_t|{\varphi }_{\theta }(s_t))\left[Q^{\mathrm{ret}}(a_t,s_t)-V_t(s_t)\right]\hfill \\ & & +\underset{a\sim \pi }{\mathbb{E}}\left({\left[\frac{{\rho }_t(a)-c}{{\rho }_t(a)}\right]}_{+}{\nabla }_{{\varphi }_{\theta }(s_t)}\log f(a_t|{\varphi }_{\theta }(s_t))\right.\hfill \\ & & \times \left.\left[Q_{{\theta }_c}(a,s_t)-V_{{\theta }_c}(s_t)\right]\phantom{\left[\frac{{\rho }_t(a)-c}{{\rho }_t(a)}\right]}\right).\hfill \end{array}$$ (6)

Finally, to limit the per‐step changes to the policy and achieve stability, ACER provides the option to utilize a modified version of trust region policy optimization (TRPO), ²⁷ ensuring that the updated policy does not deviate significantly from the average policy network ${\varphi }_a$. Specifically, it restricts the policy network parameter $\theta$ updating at step $t$ as

$$d\theta =d\theta +\frac{\partial {\varphi }_{\theta }(s_t)}{\partial \theta }\left(g_t^{\mathrm{ACER}}-\max \left\{0,\frac{k^T{g}_t^{\mathrm{ACER}}-\delta }{{||k||}_2^2}\right\}k\right).$$ (7)

Here, $k={\nabla }_{{\varphi }_{\theta }(s_t)}{D}_{KL}[f(·|{\varphi }_{{\theta }_a}(s_t))||f(·|{\varphi }_{\theta }(s_t))]$ is the linear Kullback–Leibler divergence ($D_{KL}$) and $\delta$ is the predefined divergence constraint. In this situation of disabled TRPO updating, the policy network parameter is updated as

$$d\theta =d\theta +\frac{\partial {\varphi }_{\theta }(s_t)}{\partial \theta }{g}^{\mathrm{ACER}}.$$ (8)

In practice, entropy regularization term $\beta {\nabla }_{\theta }H(\pi (s,\theta ))$ ²² can also be used to boost online and offline policy update performance, with $\beta$ the weighting parameter. This term improves exploration by discouraging premature convergence to sub‐optimal deterministic policies.

With variables $d{\theta }_c$ and $d\theta$ computed, the network parameters ${\theta }_c$ and $\theta$ are updated with RMSprop algorithm. ²⁸ Under active TRPO updating, the parameters ${\theta }_a$ for the average policy network ${\varphi }_a$ is updated as ${\theta }_a=\alpha {\theta }_a+(1-\alpha )\theta$, with $\alpha$ the average model decay rate.

2.3.2. Establishment of the VTP system upon ACER architecture

We apply the ACER architecture to develop the VTP system for automatic treatment planning as follows. We define an input state $s$ as discrete points taken from the DVH curves of a treatment plan. For a treatment plan containing $M$ target and OARs, the input state $s$ has a dimension of $m\times M$, where $m$ represents the number of discrete points on each DVH curve. The action space consists of tuning strategies for the TPPs. In this initial approach, we allow each TPP to have two tuning strategies: increasing or decreasing by a predefined amount. With $M$ target and OARs, there are $3M$ TPPs to tune, resulting in an action space of length $6M$. This defines the dimensions of both the policy distribution and the $Q$‐value function space, each with length $6M$.

Once a TPP tuning action $a$ is predicted for treatment plan $s$ at time $t$, the in‐house TPS performs inverse treatment planning to generate a new plan $s_{t+1}$. For each state‐action pair $(s_t,a_t)$, the immediate reward $r$ is calculated as the difference in plan quality between the new plan and the current plan, that is, $r_t=\psi (s_{t+1})-\psi (s_t)$, where $\psi (s)$ represents the quality evaluation of the plan $s$. The total reward is computed as $R={\sum }_{i=t}^{t_{\mathrm{epi}}-1}{\gamma }^{i-t}{r}_i$, reflecting the accumulated return across all future plans in that episode.

Training of the ACER‐based VTP follows the standard ACER training process. We launch $N_a$ parallel online training agents, each with distinct input treatment plans. The agents asynchronously update the network policy upon completing an episode with a maximum step length of $t_{\mathrm{epi}}$. After each episode, each agent is restarted with a different input treatment plan. During the process, each agent stores its episodic trajectories in a replay buffer with a storage length of $t_{\mathrm{rep}}$ steps. Once $t_s$ steps have been accumulated, offline policy training examines the experience pool using a batch size of $t_B$ steps and begins to update the policy network at a frequency $p-1$ times higher than that of the online updates. To consistently monitor training performance, we evaluate the process every $t_{\mathrm{eval}}$ steps using independent evaluation patient cases. The training continues until a maximum of $T_{\max }$ steps is reached.

2.3.3. Testbed

In line with our previous development efforts, ¹⁶ this paper continues to use IMRT treatment planning for prostate cancer as a testbed to test the proposed ACER‐based VTP system. We consider a scenario involving one target (the prostate) and two OARs (the bladder and the rectum), leading to the optimization of three DVH curves. Each curve is represented by 100 discrete points, resulting in a total of 300 floating values as the input to the ACER agent. According to Equation (1), we have 9 TPPs to tune, creating an action space of 18, as shown in Table 1. The specific increment and decrement amplitudes are determined based on experience and are not expected to significantly affect the overall convergence of the treatment planning process. F

TABLE 1.

The actions to tune the TPPs in step $t$ based on the TPP values in step $t-1$.

Actions	${\lambda }_{\mathrm{PTV},\mathrm{OAR}}^t$	$t_{\mathrm{PTV}}^t$	$t_{\mathrm{OAR}}^t$	$V_{\mathrm{PTV}}^t$	$V_{\mathrm{OAR}}^t$
Increase	$1.65{\lambda }_{\mathrm{PTV}}^{t-1},\mathrm{OAR}$	$\min (1.2$, $1.01t_{\mathrm{PTV}}^{t-1})$	$\min (1$, $1.25t_{\mathrm{OAR}}^{t-1})$	$\min (0.3$, $1.25V_{\mathrm{PTV}}^{t-1})$	$\min (1$, $1.25V_{\mathrm{OAR}}^{t-1})$
Decrease	$0.6{\lambda }_{\mathrm{PTV}}^{t-1},\mathrm{OAR}$	$\max (1$, $0.91t_{\mathrm{PTV}}^{t-1})$	$0.6t_{\mathrm{OAR}}^{t-1}$	$0.8V_{\mathrm{PTV}}^{t-1}$	$0.8V_{\mathrm{OAR}}^{t-1}$

Note: Here, “OAR” represents both rectum and bladder.

Abbreviations: OAR, organ at risk; TPPs, treatment planning parameters.

We use the ProKnow scoring system (ProKnow Systems, Sanford, FL, USA) for prostate cancer IMRT plans to estimate the plan quality $\psi (s)$. Nine original scoring criteria from the ProKnow system relevant to our testbed are shown in Table 2. In practice, we slightly adjust these criteria ¹⁴ and apply them to compute $\psi (s)$ for each plan $s$. With equal weighting on each criterion, the score ranges from 0 to 9.

TABLE 2.

The nine criteria originated from the planIQ scoring system relevant to this study.

Bladder				Rectum				PTV
$V_{100.6\%}$	$V_{94.3\%}$	$V_{88.1\%}$	$V_{81.2\%}$	$V_{94.3\%}$	$V_{88.1\%}$	$V_{81.2\%}$	$V_{75.5\%}$	$D_{0.03cc}$
$<20\%$	$<30\%$	$<40\%$	$<55\%$	$<20\%$	$<30\%$	$<40\%$	$<55\%$	$<110\%$

For this testbed, the specific network architecture of the ACER‐based VTP system is shown in Figure 2. The input DVH data pass through a fully connected layer with a hidden size of 32, followed by a Rectified Linear Unit (ReLU) activation layer. They then enter a long short‐term memory (LSTM) layer with a hidden size of 32. Afterward, the data are split: one path goes through a fully connected layer followed by a softmax layer, which outputs the policy, while the other path flows into a separate fully connected layer to produce the $Q$ value function. The implementation of the network relies on the pytorch and Open AI Gym libraries. The learnable parameters are distributed across the three fully connected layers and the LSTM cell, totaling around 20,000 parameters.

FIGURE 2.

The deep neural network featuring a “two‐head” structure designed for the ACER‐based VTP. It takes the DVH of the current plan as input. It outputs a TPP tuning strategy across 18 actions in one head, and produces the corresponding Q‐value in the other head. ACER, actor critic with experience replay; DVH, dose volume histogram; TPP, treatment planning parameter; VTP, virtual treatment planner.

2.3.4. Datasets

We use three independent datasets of prostate cancer IMRT cases for network training, validation and testing. The first dataset contains 52 independent patient treatment plans, as described in our previous development. ¹³ , ¹⁶ The second dataset is the Common Optimization for Radiation Therapy (CORT) dataset from the Mass General Radiation Oncology Physics Division, which is publicly available and contains one independent patient case. ²⁹ The third dataset is The Radiotherapy Optimization Test Set (TROTS), which includes 30 independent patient cases. ³⁰

The three datasets differ in anatomical detail and available beam configurations. Datasets 1 and 2 provide dose deposition matrices from 180 beam angles (every 2 degrees) covering the entire PTV and OAR volumes. In contrast, Dataset 3 includes dose deposition matrices sampled at specific points within the PTV and OAR volumes for 25 unique combinations of gantry, couch, and collimator angles (Table 3). In this study, we selected a subset of beam angles from the 180 available angles for patient cases in Datasets 1 and 2, while utilizing all 25 available beam configurations from Dataset 3 for network training and testing. Regarding anatomical differences, Dataset 1 generally has higher voxel counts for the PTV, bladder, and rectum compared to Datasets 2 and 3, whereas Dataset 2 has the largest bladder‐to‐PTV overlap. Since only sampled anatomical volumes are available in Dataset 3, it has the smallest voxel counts, especially for OARs, as well as the least overlap between OARs and the PTV. Detailed statistics, including voxel counts for the PTV and OARs, their relative overlaps, and the number of beamlets for each dataset and beam configuration, are summarized in Table 4. Overall, these datasets collectively provide a rich and diverse resource for training and evaluating our ACER‐based agent.

TABLE 3.

Beam settings (gantry, couch, and collimator angles in degrees) for dataset 3.

Gantry	12.1	20.6	20.9	21.7	43.1	43.1	54.6	58.3	66.2	78.1	90.0	95.9	258.4
Couch	330.5	61.7	0.0	90.0	17.2	342.8	19.3	0.0	339.1	339.1	0.0	328.8	0.0
Collimator	88.8	24.4	0.0	0.0	4.2	8.5	24.7	89.1	4.1	8.1	25.7	113.0	17.2
Gantry	264.1	281.6	281.9	288.0	293.5	293.8	298.2	298.2	309.6	309.6	316.9	347.9
Couch	31.2	0.0	20.9	349.9	43.3	20.9	10.8	349.2	7.6	352.4	342.8	29.5
Collimator	29.1	69.3	8.1	41.3	13.4	87.9	41.3	55.9	37.8	17.2	16.9	69.0

TABLE 4.

Voxel and beamlet numbers, as well as organ overlap percentages, for the training patient case and the test patient cases in Datasets 1, 2, and 3.

		Testing
	Training	Dataset1	Dataset2	Dataset3
${\mathrm{V}}_{\mathrm{PTV}}$	7466	$9266\pm 3002$	6770	$4954\pm 18$
${\mathrm{V}}_{\mathrm{BLA}}$	1513	$16506\pm 11529$	11592	$4975\pm 11$
${\mathrm{V}}_{\mathrm{REC}}$	2026	$4396\pm 1343$	1760	$4944\pm 27$
${\mathrm{V}}_{\mathrm{BLA}}\cap {\mathrm{V}}_{\mathrm{PTV}}/{\mathrm{V}}_{\mathrm{PTV}}$ (%)	4.4	$7.0\pm 3.1$	25.7	$0.1\pm 0.1$
${\mathrm{V}}_{\mathrm{REC}}\cap {\mathrm{V}}_{\mathrm{PTV}}/{\mathrm{V}}_{\mathrm{PTV}}$ (%)	3.9	$1.7\pm 1.1$	1.7	$0.0\pm 0.0$
${\mathrm{N}}_{\mathrm{Beamlets}}$ (7 beams)	2800	$2742\pm 807$	948	N/A
${\mathrm{N}}_{\mathrm{Beamlets}}$ (new 7 beams)	N/A	$2912\pm 652$	996	N/A
${\mathrm{N}}_{\mathrm{Beamlets}}$ (6 beams)	N/A	$2496\pm 559$	848	N/A
${\mathrm{N}}_{\mathrm{Beamlets}}$ (25 beams)	N/A	N/A	N/A	$2937\pm 333$

For the specific training, validation, and testing setup, we trained our model using one patient case from Dataset 1 with beam angles set at 0, 32, 64, 96, 264, 296, and 328, and validated the model using two additional independent patient cases from Dataset 1 with the same angle configuration. Six different test groups were created for testing, as summarized in Table 5. Test group 1 comprises 49 patient cases from Dataset 1 that are independent of the training and validation sets. For this group, we used fixed TPP initialization (all values set to 1, except for $V_{\mathrm{PTV}}=0.1$) to generate the initial treatment plans, consistent with the initialization approach from the previous DQN‐based treatment planning study, ¹⁶ thus allowing direct comparison of performance between the two network architectures. Test groups 2 through 6 were generated using random TPP initialization, motivated by two main considerations: first, random initialization enables testing across diverse state‐action‐reward trajectories by creating a wide variety of initial treatment plans; second, it simulates variations in TPP settings across different radiation oncology clinics.

TABLE 5.

Summary of the testing groups, including the number of plan cases, datasets, beam configurations, and TPP initialization methods.

Test Group	${\mathrm{N}}_{\mathrm{Plans}}$	Dataset	Beam Configuration	TPP Initialization
1	49	1	Same as training setup	Fixed
2	147	1	Same as training setup	Random
3	30	2	Same as training setup	Random
4	30	1 & 2	new 7‐beam (26, 76, 130, 180, 230, 286, 336)	Random
5	30	1 & 2	6‐beam (0, 60, 120, 180, 240, 300)	Random
6	90	3	Dataset 3 specific	Random

Abbreviation: TPP, treatment planning parameter.

2.4. Adversarial attack

After completing the training of the ACER agent, except for testing its performance on TPP tuning decisions, we also evaluate its robustness against adversarial attacks. Adversarial attacks are malicious attempts to manipulate machine learning models into making incorrect predictions or decisions. ³¹ , ³² Given the potential clinical application of automatic treatment planning agents in the future, it is crucial to train the ACER agent to be robust against such attacks. ⁹

We assume the adversary has access to the trained policy network, allowing it to fool the network by perturbing the input state in a way that exploits the policy's sensitivity. Specifically, we use the fast gradient sign method (FGSM) ³³ to compute the state perturbation $\eta$, which is the gradient of the loss function $J$ with respect to the state $s$:

$$\eta =\epsilon \mathrm{sign}({\nabla }_{s}J(\theta ,s,y)).$$ (9)

Here, $\epsilon$ is a constant, serving as the upper limit for the element‐wise perturbation, i.e., ${|\eta |}_{\infty }\le \epsilon$. $y$ is the distribution over all possible actions. With this, the perturbed state becomes $s^{\prime }=s+\eta$.

We apply the FGSM‐based attack to both the ACER agent trained in this work and the DQN agent from our previous study, ¹⁶ and compare their robustness to the attack. In the ACER agent, $y$ is the stochastic policy $\pi$. $J(\theta ,s,y)$ is represented as the cross entropy loss between $y$ and the distribution that places all weight on the highest‐weighted action $a_j$ in $y$. Specifically

$$J(\theta ,s,\pi )=-\frac{1}{18}\left(\log ({\pi }^{\theta }(a_j|s))+\sum _{i\ne j}\log (1-{\pi }^{\theta }(a_i|s))\right).$$ (10)

In the DQN agent, the policy determined by the Q value function is deterministic, which causes the problem that the gradient of $J(\theta ,s,y)$ is almost zero for all input states. To solve it, we define $y$ as the softmax of the $Q$ value function. ¹⁹ We set $\epsilon$ to be values of 0.001, 0.01, and 0.1, apply the corresponding attacks and record their perturbations on action priorities and the next‐step treatment plan qualities.

3. RESULTS

3.1. Training results

The hyperparameter values used to configure the ACER‐based VTP for prostate cancer IMRT treatment planning are listed in Table 6. We utilize 3 CPU cores for online asynchronous training, aiming to complete the entire training within 250,000 steps. To encourage early convergence in the treatment planning process, we limit the length of each episode to 20 steps. Offline training begins with the storage of 2000 steps of experiences, with a total maximum experience storage capacity of 100,000 steps. The TRPO‐based updating method has been found to be computationally expensive; therefore, we disable it in this study to simplify the network. The values for the remaining parameters in Table 6 are consistent with those used for training the ACER agent in the Atari57 game set. ²⁰ The entire training process takes approximately 7 days on an Intel(R) Core(TM) i7‐6850K CPU @ 3.60GHz.

TABLE 6.

The hyperparameters and their values used to train the ACER‐based VTP for IMRT treatment planning of prostate cancer.

Hyperparameter	Value	Description
$N_a$	3	Number of asynchronous training agents
$T_{\max }$	250,000	Number of total training steps
$t_{\mathrm{epi}}$	20	Maximum length of an episode
$t_{\mathrm{rep}}$	100,000	Storage capacity of experience replay memory
$t_s$	2000	Number of accumulated transitions before starting off‐policy training
$t_B$	16	Off‐policy batch size
$c$	10	Importance‐weight truncation in experience replay
$p$	4	Ratio of off‐policy to on‐policy updates
$t_{\mathrm{eval}}$	500	Interval between two adjacent evaluations
$\gamma$	0.99	Discount factor
$\beta$	0.001	Weighting for entropy regularization
$\alpha$	disabled	Decay rate for the average policy model
$\delta$	disabled	Trust region threshold value

Abbreviations: ACER, actor critic experience replay; IMRT, intensity modulated therapy; VTP, virtual treatment planner.

The convergence map of the agent training process, evaluated based on the average plan score for the validation patient cases, is shown in Figure 3. As illustrated, the plan score gradually approaches the maximum value of 9 as training progresses, with reduced fluctuations until approximately 200,000 steps. Beyond this point, performance becomes unstable, exhibiting large fluctuations, which we interpret as overfitting to the training cases. Therefore, we select the policy obtained at an earlier convergence point, around step 120,500, for testing.

FIGURE 3.

The convergence map of the ACER‐based virtual treatment planner VTP training process, evaluated based on the average plan score for the validation patient cases. ACER, actor critic with experience replay; VTP, virtual treatment planner.

3.2. Testing results

The results for test group 1 are shown in Figure 4a,c. In Figure 4a, the patient cases are grouped into eight categories based on their plan scores (group index $i\in (1,2,\text{…},8)$ includes cases with plan scores in the range of $[i,i+1)$). Before ACER‐guided treatment planning, plan scores are distributed broadly from 2 to 9, with a mean score and standard deviation (std.) of $6.20\pm 2.01$. After ACER‐guided treatment planning, 42 out of 49 cases achieve a full score of 9, 1 case reaches 8.9, 3 cases score 8, and 3 cases score between 7 and 8. The corresponding mean and std. are $8.85\pm 0.43$. In comparison, DQN‐guided treatment planning improves the plan score from $6.18\pm 1.75$ to $8.14\pm 1.27$ for 50 patient cases from the same patient dataset. ¹⁶

FIGURE 4.

(a,b) The plan score distributions for 49 test cases generated under trivial treatment planning parameter (TPP) settings and 147 test cases generated under random TPP settings from 49 patient cases in dataset 1, respectively, before and after actor critic with experience replay (ACER)‐guided treatment planning. The histogram width is set to 1. (c,d) The mean and standard deviation of the plan score distributions before and after ACER‐based treatment planning for the cases shown in (a) and (b), respectively. The groups in (c) and (d) correspond one‐to‐one with the histogram distributions in (a) and (b). ACER‐based treatment planning significantly improves plan quality, achieving a mean score close to 9 across all plan groups. ACER, actor critic with experience replay, TPP, treatment planning parameter.

In Figure 4c, the same 49 patient cases are divided into 8 groups based on their initial plan scores. The mean and standard deviation of the plan scores for each group, both before and after ACER‐based treatment planning, are plotted. It is evident that after ACER‐guided treatment planning, the plan scores are uniformly improved, approaching 9 across all patient groups, including those with very low initial scores below 3 (patient group 1). In contrast, DQN‐based treatment planning shows that some patients with low initial scores could not be efficiently improved (as depicted in Figure 5a in Sprouts et al. ¹⁶ )

FIGURE 5.

(a–c) The dose colorwash for a representative test case in step 0, step 10, and final step 16, respectively, under actor critic with experience replay (ACER)‐based automatic treatment planning. The contours are represented in black for the prostate target, blue for the rectum, and green for the bladder. In the color bar, color “1” corresponds to the prescription dose level. (d) The corresponding dose volume histograms (DVHs) for steps in (a–c). (e) The TPP tuning choices made by ACER for each planning step. (f) The corresponding plan scores over the 16 steps (the maximum score is 9). ACER, actor critic with experience replay; TPP, treatment planning parameter.

We illustrate how ACER‐based VTP observes an intermediate treatment plan and makes the TPP adjustment decision for a representative testing case in Figure 5. As is shown in Figure 5a,d, at the initial step, the plan fails to spare the bladder, partially fails to spare the rectum, and has a hotspot in the PTV, resulting in a low initial plan score of 2. The VTP observes this plan and decides to lower the threshold dose value for the bladder ($t_{\mathrm{BLA}}$) in the first step, which improves the plan score to 4 by fully sparing the bladder volume. It then continues to enhance rectum sparing by lowering the threshold value for the rectum ($t_{\mathrm{REC}}$), However, these adjustments result in an even hotter PTV. To address this issue, over the next 14 steps, the ACER‐based VTP reduces the priorities for the OARs (${\lambda }_{\mathrm{REC}}$ and ${\lambda }_{\mathrm{BLA}}$), lowers threshold dose value in PTV ($t_{\mathrm{PTV}}$), and increases the PTV priority (${\lambda }_{\mathrm{PTV}}$) until reaching a score of 9. These actions relax OAR constraints while tightening PTV constraints, mirroring the adjustments a human dosimetrist would make. This indicates that the ACER‐based agent exhibits a human‐like approach to TPP tuning.

We then expanded our evaluation to test groups 2 through 6, highlighting variations in random TPP initialization, distinct beam configurations, and differences in patient anatomy compared to the training set and test group 1. Representative treatment planning cases are presented in Figure 6, and statistical results are summarized in Table 7. Additionally, results for test group 2 are shown in histogram format in Figure 4b,d to provide a side‐by‐side comparison with test group 1.

FIGURE 6.

Representative treatment planning cases from the training and test groups. The first three columns show dose color wash maps from the initial treatment plan, an intermediate step, and the final treatment plan generated by the automatic treatment planning process guided by the ACER‐based agent. The last column presents the corresponding DVHs for these three steps. ACER, actor‐critic with experience replay; DVHs, dose‐volume histogram.

TABLE 7.

Statistics of the treatment planning results for the six test groups (Table 5).

Test groups	${\mathrm{Score}}_i$	${\mathrm{Score}}_f$	${\mathrm{P}}_9$ (%)	${\mathrm{N}}_{\mathrm{Steps}}$
1	6.2 $\pm$ 2.0	8.9 $\pm$ 0.4	85.7	18.8 $\pm$ 7.3
2	6.3 $\pm$ 1.6	8.9 $\pm$ 0.3	91.8	19.2 $\pm$ 5.5
3	3.9 $\pm$ 0.3	9.0 $\pm$ 0.0	100.0	18.9 $\pm$ 2.4
4	4.2 $\pm$ 1.0	9.0$\pm$ 0.0	100.0	18.6 $\pm$ 4.5
5	4.2 $\pm$ 1.0	8.8 $\pm$ 0.4	80.0	21.9 $\pm$ 5.4
6	8.0 $\pm$ 0.1	9.0 $\pm$ 0.2	95.6	16.4 $\pm$ 6.7

Note: Reported are mean $\pm$ std of initial (${\mathrm{Score}}_i$) and final (${\mathrm{Score}}_f$) treatment plan scores, percentage reaching full plan score (${\mathrm{P}}_9$), and mean $\pm$ std of planning steps (${\mathrm{N}}_{\mathrm{Steps}}$).

In Figure 4b, the patient cases are grouped into 7 categories based on their plan scores, using the same method as in Figure 4a. Compared to the initial treatment plans generated under fixed TPP initialization (Figure 4a), treatment plans generated using random TPP initialization exhibit an obviously different score distribution. Before ACER‐guided treatment planning, the mean and std. of the treatment plans are $6.33\pm 1.65$. After planning, 135 out of 147 cases achieve a full score of 9, and 12 cases reach a score of 8, with no cases scoring below 8. The corresponding mean and std. are $8.92\pm 0.27$. The corresponding patient group distribution is shown in Figure 4d, which also shows that ACER‐guided treatment planning improve the plan score uniformly across all patient groups.

In test group 3, characterized by significantly higher bladder‐to‐PTV overlap compared to the training cases (Table 4), the initial plan scores are notably low, averaging $3.9\pm 0.3$. Despite this challenge, the ACER‐based agent successfully elevated the treatment plan scores to the maximum value of 9 for all 30 test cases.

In test groups 4 and 5, we specifically evaluated the ACER agent's performance on automatic treatment planning with beam‐angle configurations different from those used in training. For each group, initial treatment plans were generated by randomly selecting 15 patient cases from Dataset 1 (one plan per patient) and generating 15 additional plans from Dataset 2, resulting in an average initial plan score of $4.2\pm 1.0$ across both groups. The ACER agent performed robustly with the new 7‐beam configuration (test group 4), successfully elevating all cases to a score of 9. However, it faced slight challenges with the six‐beam configuration (test group 5), likely due to the intrinsic complexity of optimizing fewer beam angles. Although test group 5 represented the lowest performance among the test groups, the agent still managed to elevate 10 out of 15 cases from Dataset 1 and 14 out of 15 cases from Dataset 2 to a score of 9. The remaining cases achieved a score of 8, with the exception of one case from Dataset 1, which attained a score of 7.34.

Lastly, test group 6 emphasizes substantial differences in beam configurations and anatomical characteristics compared to both the training cases and previous test groups. In this group, initial plans frequently achieved a score of 8, primarily due to the relatively small overlap between OARs and the PTV. Although all OARs were adequately spared, the PTV dose distributions were generally hotter than those observed in other datasets. Following ACER‐guided treatment planning, the network successfully elevated 95.6% of these cases to a score of 9.

Combing all test cases, the mean $\pm$ std. of the plan score distributions before ACER‐based treatment planning is $6.17\pm 1.90$. After implementing ACER‐based treatment planning, $92.29\%$ of the cases achieve a perfect score of 9, with only $6.65\%$ scoring between 8 and 9, $0.01\%$ scoring between 7 and 8, and no cases scoring below 7. The mean $\pm$ std. of the final scores is $8.92\pm 0.29$.

For all testing groups, the network demonstrates consistent performance regarding the number of TPP tuning steps required, with only a slight increase observed in more challenging planning scenarios (e.g., test group 5) (Table 7). This highlights the stability and robustness of the trained ACER‐based agent in automatic treatment planning. However, differences in dose deposition matrix sizes lead to varying computation times during the inverse optimization process performed by the TPS. At each step, the ACER‐based VTP agent rapidly selects an action upon receiving DVH input, typically on the order of ${10}^{-4}$ seconds. Nonetheless, the inverse treatment planning time under a given set of TPPs currently ranges from 3 to 28 seconds using our in‐house TPS. We anticipate that integrating this VTP with a more efficient TPS could significantly enhance the overall efficiency of the automatic treatment planning process.

Finally, since the ACER‐based treatment planning agent utilizes a stochastic policy, it is important to understand its policy behavior to ensure stability in guiding treatment planning. To investigate this, we identify five common cases from ACER‐guided and DQN‐guided treatment plannings, each with an initial plan score of 5 due to the failure in rectum dose sparing. The mean and standard deviation of the policy distributions and Q‐value distributions for the leading 18 actions in these cases are shown in Figure 7a,b, respectively. As expected, both networks exhibit relatively stable preferences for certain actions when faced with similar input plans. However, it is noteworthy that ACER strongly prioritizes the reasonable action for rectum sparing, that is to “decrease $t_{\mathrm{REC}}$,” with a magnitude of order higher than other actions. It also significantly suppresses non‐reasonable actions like “increase $t_{\mathrm{REC}}$” or “decrease ${\lambda }_{\mathrm{REC}}$,”, which are several orders of magnitude lower than the leading action. The sub‐leading actions, such as “decrease $t_{\mathrm{BLA}}$” or “increase ${\lambda }_{\mathrm{REC}}$,” are either reasonable or have unclear effects. This type of policy distribution allows the agent to explore the action space while maintaining effectiveness. In contrast, DQN does not significantly differentiate between reasonable and non‐reasonable actions. In fact, the leading two actions in DQN have contradictory effects on rectum dose sparing. This further highlights the superior performance of the ACER‐based VTP agent.

FIGURE 7.

Policy distributions from the actor critic with experience replay (ACER) agent (top row) and Q value distributions from the deep Q network (DQN) agent ¹⁶ (bottom row) for five patient cases with similar plan qualities. In all patient cases, the plans fail to spare the rectum with losing all 4 credits for the 4 rectum dose‐volume criteria as shown in Table 2. ACER, actor critic with experience replay; DQN, deep Q network.

3.3. Adversarial attack

We randomly choose 30 treatment plans and apply one FGSM attack under each value of $\epsilon$ for both ACER and DQN networks. With three $\epsilon$ values of 0.001, 0.01, and 0.1, a total of 180 attacks are performed. The results are shown in Figure 8 and Table 8.

FIGURE 8.

The illustration of the attack effect from FGSM on the DQN ¹⁶ and the ACER agent based treatment planning. (a) The DVH distributions before and after the perturbation with $\epsilon =0.001$. The corresponding DVH distributions generated by the in‐house TPS under the tuned TPPs by DQN (b) and ACER‐based agent (c) before and after the perturbation. ACER, actor critic with experience replay; DQN, deep Q network; DVH, dose volume histogram; FGSM, fast gradient sign method; TPP, treatment planning parameter; TPS, treatment planning system.

TABLE 8.

The statistical performance of ACER‐ and DQN‐based agents under three levels of adversarial attacks.

		ACER	DQN
$\epsilon =0.001$	success rate	$0.0\%$	$30.0\%$
	$\mathrm{\Delta }{P}_1$	$-0.194\%\pm 0.25\%$	$-18.55\%\pm 7.73\%$
$\epsilon =0.01$	success rate	$3.3\%$	$100\%$
	$\mathrm{\Delta }{P}_1$	$0.08\%\pm 10.24\%$	$-87.95\%\pm 17.98\%$
$\epsilon =0.1$	success rate	$53.3\%$	$93.3\%$
	$\mathrm{\Delta }{P}_1$	$-36.81\%\pm 46.49\%$	$-68.99\%\pm 68.94\%$
$\mathrm{\Delta }{P}_{12}$		$67.08\%\pm 21.30\%$	$7.19\%\pm 5.19\%$

Abbreviations: ACER, actor critic with experience replay; DQN, deep Q networks.

An illustration of the attack effect with perturbation to the input states at the $\epsilon =0.001$ level is shown in Figure 8. Figure 8a shows the DVH distributions before and after the perturbation with $\epsilon =0.001$, which is not distinguishable by naked eyes. The initial plan has a score of 3, which partially fails to spare the rectum and the bladder (both lose 3 points following criteria in Table 2). Before the perturbation is applied, DQN produces a $Q$‐value distribution that maximizes the action “decrease $t_{\mathrm{REC}}$,” which improves the plan score to 4. However, after the perturbation, the Q value for the initial top action is reduced by $16.52\%$, and the optimal action changes to “increase ${\lambda }_{\mathrm{REC}}$,” slightly decreasing the plan score to 2.26. The treatment plans generated under original action and perturbation changed action are shown in Figure 8b. In contrast, for the same patient case, ACER's policy distribution prioritizes “decreasing $t_{\mathrm{BLA}}$” and “decrease $t_{\mathrm{REC}}$” as the top two leading actions, with probabilities of 0.86 and 0.19, respectively (the sum of all action probabilities is 1). The ACER‐based stochastic policy selects “decrease $t_{\mathrm{BLA}}$,” which improves the plan score to 5. After the perturbation is applied, the policy distribution retains the same ranking of actions, with the probabilities of the leading two actions changing by $-0.05\%$ and $0.35\%$, respectively. The new treatment plans generated are shown in Figure 8c. This demonstrates ACER's stable performance under adversarial attack.

The statistical results for all 180 attacks are shown in Table 8. As the perturbation level increases from $\epsilon =0.001$ to $\epsilon =0.1$, the attack success rate increases from $0.0\%$ to $53.3\%$ for the ACER agent, while it rises from $30.0\%$ to $93.3\%$ for the DQN agent. This indicates that the ACER network demonstrates greater robustness compared to the DQN. To further understand the effects of the attacks, we analyzed changes in action probabilities for ACER and Q values for DQN under various attack levels. We compared these with the differences in probabilities and Q values between the top two leading actions. The analysis reveals that the probability difference between the top two leading actions averages at $67.08\%\pm 21.30\%$ for the ACER agent. Comparing to it, the mean probability change for the leading action in ACER is negligible at $\epsilon =0.001$ and 0.01 levels. In contrast, for the DQN agent, the mean Q value changes at all perturbation levels are significantly greater than the Q value difference between the top two leading actions. This behavior further explains the relative robustness of the ACER agent against FGSM attacks.

4. DISCUSSION

In summary, we have developed an ACER‐based VTP agent that can effectively guide the in‐house TPS for inverse treatment planning with prostate IMRT patient cases as testbed. Despite being trained on a single patient case with random initializations of TPPs, the network has demonstrated superior test performance compared to our previously developed DQN‐based VTP agent. ¹⁶ It has also demonstrated strong generalization, performing well on data from distinct resources, and has exhibited stable performance under adversarial attacks. This raises the question: why does ACER perform so much better?

First, ACER is stochastic policy based, allowing it to explore a wider solution space without suffering from scaling issues typically associated with DQN. The application of the entropy regularization term during the policy update promotes global convergence. Additionally, ACER incorporates various strategies for effectively managing bias and variance during policy updates. These attributes likely contribute to the network's improved performance in treatment planning. Further investigations are needed to explore how these features impact the network's efficacy.

Second, the greater robustness of the ACER agent compared to the DQN agent in the FGSM attack can be understood from two perspectives. First, the stochastic policy strategy of ACER contributes to better convergence compared to DQN. This aligns with findings that show A3C outperforms DQN in adversarial attacks on Atari games. ¹⁹ Second, unlike tasks such as Atari, which typically have only one optimal action, inverse treatment planning often involves multiple reasonable TPP tuning actions that can enhance plan quality. A well‐trained ACER network that effectively prioritizes these reasonable TPP tuning strategies over less suitable options can contribute to its stable performance, even under adversarial attacks.

In this study, we utilize an in‐house TPS for inverse treatment planning during ACER agent training, which may raise concerns about the ACER agent's ability to operate a commercial TPS for high‐quality treatment planning. However, we have previously demonstrated the high efficacy of a DRL agent trained with our in‐house TPS in operating commercial TPS to achieve high‐quality treatment planning. ¹⁶ This suggests that the current ACER‐based agent, trained with the same in‐house TPS, can also function effectively with commercial TPS. We plan to evaluate the performance of the ACER agent on commercial TPS in future work.

Additionally, we use a relatively simple plan quality evaluation system in developing the ACER agent, which may result in differences in plan quality compared to those assessed under clinical evaluation systems. However, we want to emphasize that the primary goal of this work is to demonstrate the efficacy of the ACER agent in treatment plan parameter tuning under a specific reward system. We do not anticipate convergence issues under different evaluation systems.

Another limitation is that we utilize the computationally‐efficient FGSM adversarial attack to test the robustness of ACER‐based VTP agent. Although this method has shown effectiveness in fooling A3C network on other tasks, ¹⁹ it may not be effective in attacking this VTP agent where multiple suitable actions exist. In our future work, we will test the robustness of this VTP agent by employing stronger adversarial attacks.

Finally, in this initial study, we employ the ACER network with a discretized action space, consistent with our previous DQN‐based development efforts. However, to make this tool practical for treatment planning in real clinical settings, it is essential to train a DRL agent capable of tuning each TPP in a continuous space, and ideally, tuning multiple TPPs in a single planning step. Based on the results of this study, we have identified the potential of ACER to produce a prioritized TPP tuning space that could facilitate multi‐TPP tuning in one step. To advance this approach, we may need to further optimize the TPP tuning space to prioritize reasonable TPP strategies effectively. This could be achieved by enhancing the entropy regularization term, activating the TRPO strategy, and enriching the training datasets. Additionally, ACER has a continuous‐action counterpart that can be used to explore for continuous TPP tuning. These will be our next step work in the future.

This also raises another important issue worth exploring: the TPP tuning hyperspace. Since the beginning of inverse treatment planning, this hyperspace has remained largely unexplored, with dosimetrists relying on intuition and experience to navigate it. A DRL‐based VTP could effectively map this hyperspace and generate insights that dosimetrists can learn from, as illustrated by the prioritized actions derived from the ACER policy distribution. We believe that further investigation into the continuous and simultaneous tuning of multiple TPPs using advanced DRL strategies can not only enhance automation in real‐time treatment planning but also contribute to a deeper understanding of the TPP tuning hyperspace. Much like how DRL algorithms have revolutionized game strategies, like in chess, a well‐designed DRL agent has the potential to reshape conventional approaches to TPP tuning, ultimately guiding the radiotherapy clinics toward more effective treatment planning.

5. CONCLUSION

We have trained a deep reinforcement network with actor–critic experience replay technique for automatic treatment planning. The trained network can guide the in‐house treatment planning system for high quality treatment planning when using the prostate cancer IMRT as a testbed. The trained network has high generality, which performs well over patient data from sources distinct from the training patient dataset. It also shows high robustness over adversarial attack, demonstrating its potential in practical treatment planning in clinical settings.

CONFLICT OF INTEREST STATEMENT

The authors declare no conflicts of interest.

Supporting information

Supporting Information

Abrar MM, Sapkota P, Sprouts D, Jia X, Chi Y. Actor critic with experience replay‐based automatic treatment planning for prostate cancer intensity modulated radiotherapy. Med Phys. 2025;52:e17915. 10.1002/mp.17915