Automated deep learning by recurrent hyperparameter optimization

Abstract

Optimizing hyperparameters of deep learning models for specific tasks requires substantial domain expertise and computational resources, remaining challenging in automated deep learning. Existing hyperparameter optimization (HPO) methods are restricted to limited discrete hyperparameter types, rely on manual priors, and fail to scale to large datasets. This paper presents Rocket, a recurrent HPO framework that automates the tuning of mixed-type hyperparameters by self-play reinforcement learning, requiring no prior domain knowledge. A policy agent is developed to learn from historical experience and progressively refine its strategy through iterative interactions with the target model. To address severe reward delay on large-scale datasets, a reward approximation mechanism is designed for data subsets, accelerating policy learning by up to 80X. Across 8 deep learning tasks and 32 benchmarks, Rocket enables target models to achieve state-of-the-art performance from scratch, matching expert-tuned results. In real industrial deployment, Rocket reduces optimization time by 13.4-fold and cost by 73%.

Rocket introduces a self-play RL framework for automated hyperparameter optimization, handling mixed types without priors. It scales large datasets via reward approximation, achieving expert-level performance while cutting time and cost in real-world deployments.

Introduction

Hyperparameter optimization (HPO) constitutes a fundamental task in automated deep learning (AutoDL)^1,2, and greatly influences model performance through automated search for optimal hyperparameter configurations (HP Configs for short. In the sequel, an HP Config denotes a collection of hyperparameters that are designated for tuning. To elaborate, an HP Config forms a complex high-dimensional space, which typically encompasses a mix of continuous variables (e.g., learning rate ranging from 1e−5 to 1e−1 and weight decay from 1e−6 to 1e−2) and discrete choices such as optimizer type. However, existing methods have largely focused on optimizing all parameters in a discretized manner, overlooking the continuous nature of some parameters.

Early grid-search methods^3–5 are fundamentally limited by the curse of dimensionality, as the number of required evaluations grows exponentially with the number of hyperparameters. Bayesian optimization (BO) techniques, such as sequential model-based optimization (SMBO)⁶, tree-structured Parzen estimators (TPE)⁷, and their successors^8,9, aim to construct a probabilistic surrogate model to approximate performance within a bounded search space. Similarly, evolutionary algorithms^10–12 have been applied to optimizing discrete architectural hyperparameters for neural network design. Hybrid approaches, including BOHB¹³, DEHB¹⁴, and BO+EVO¹⁵, seek to combine the strengths of BO and evolutionary search for improved efficiency. While these methods demonstrate utility in constrained settings, they share significant limitations: restriction to heavily discretized or low-dimensional search spaces, a pre-defined hyperparameter grid, and poor scalability to large datasets.

In this context, reinforcement learning (RL) has been introduced for HPO. Early attempts^16,17 employ policy-based learning (e.g., Proximal Policy Optimization, PPO¹⁸) and recurrent networks (e.g., LSTMs) to achieve combinatorial optimization over discrete hyperparameter sets for neural architecture search. To improve the efficiency of HPO, model-based reinforcement learning (MBRL)¹⁹ introduces a predictive surrogate model to estimate performance without full training runs. Subsequent works^20–24 adopt value-based Q-learning²⁵ frameworks to search for optimal or near-optimal discrete hyperparameter configurations. However, the value-based methods usually rely on training a separate value model. In HPO tasks where reward signals are sparse and computationally expensive, training a stable value estimator is notoriously inefficient. Besides, these methods typically operate on prior-defined, discrete hyperparameter sets, updating policies via table lookups rather than continuous optimization. For instance, the learning rate may be limited to a coarse set of values (e.g., 0.0001, 0.001, 0.01, 0.1), which lacks both granularity and justification. This discrete, tabular paradigm results in a fundamental mismatch with the high-dimensional, continuous nature of some practical hyperparameters, hindering real-world effectiveness. Moreover, when applied to large-scale datasets, previous methods remain impeded by prohibitive computational costs, as they typically necessitate exhaustive validation on the full data.

Recent advances in large language models (LLMs) have revolutionized various domains, including human-computer interaction²⁶ and complex reasoning tasks²⁷. Such progress has also inspired new research momentum in AutoDL, showing remarkable potential for enhancing deep learning pipelines (e.g., the cascade of data processing, model training, and deployment)^28–34, where the critical issue remains hyperparameter optimization. Several LLM-based works^30,33,34 have tried to utilize general-purpose LLMs (e.g., GPT-4³⁵) as black-box hyperparameter generators for producing various kinds of hyperparameters. For example, AgentHPO³⁰ employs an iterative experimental framework, where hyperparameters are refined based on pre-generated results. It operates via ChatGPT-based in-context learning, relying heavily on the prior knowledge embedded in ChatGPT, leading to poor performance on rare or unseen tasks, and posing risks of data leakage and irreproducibility. AutoMMLab³¹ enhances its end-to-end machine learning pipeline automation by introducing HPO-LLaMA, a supervised fine-tuned (SFT) model for hyperparameter generation, which relies on a large amount of real-world experiences for training. For the SFT-based methods, there exists a classic paradox regarding the precedence between “chickens” (HPO models) and “eggs” (lots of manually created hyperparameter experiences). Besides, SFT approaches predominantly memorize or recombine historical configurations rather than engage in genuine exploratory optimization. This situation poses a fundamental challenge — how to develop an HPO system without relying on empirical priors, while being suitable for various tasks?

In response to these challenges, this paper introduces Rocket, a recurrent hyperparameter optimization framework capable of effectively learning an HPO agent without relying on prior knowledge — such as empirically configured discrete hyperparameter grid or manually created tuning experiences — while autonomously guiding the training of target models from scratch. Inspired by recent advances of reinforcement learning^18,36,37, Rocket learns an LLM-based policy model (named as “HPO agent” in this paper) to generate high-dimensional, mixed-type actions (i.e., HP Configs). Then, the action rewards are derived from the performance and HP Configs of the target task to update the policy agent. As illustrated in Fig. 1, Rocket operates in an iterative and self-improving reinforcement learning fashion. Note that by optimizing the policy distribution directly through an LLM agent, Rocket treats HPO as a sequence generation problem, enabling the native generation of precise, continuous floating-point values without pre-defined grids. This mechanistic shift resolves the limitations of infinite action spaces that constrain previous value-based approaches.

Fig. 1. An overview of the proposed recurrent hyperparameter optimization framework Rocket.

During the training stage t, the HPO agent optimizes through recurrent experience replay with reward-based learning and updates the experience pool in preparation for stage t + 1. r_g, r_dif, and r_dir are reward terms derived from target model’s performance and HP Configs. After reinforcement learning, Rocket generates candidate hyperparameter configurations for training the target model.

Concretely, in the initialization phase, Rocket launches the learning process with some randomly generated synthetic experiences (or hyperparameters). The initialization aims solely to teach the agent the correct output format, so the synthetic experiences need not reflect real-world data and can be randomly generated via automated scripts at little cost. The policy agent is trained to generate diverse hyperparameter actions, with the task model’s performance and the corresponding HP Configs as well as the relationships between them serving as the optimization reward signal. Correspondingly, the reward function comprises the following components. The primary component is performance feedback — using the performance of the target model as a reward signal. Additionally, two auxiliary reward terms are incorporated based on hyperparameter relationships: (1) Hyperparameter evolving trend: This term exploits historical experiences to accelerate policy convergence towards superior hyperparameter regions. (2) Hyperparameter differentiation: This term promotes new hyperparameter exploration while preventing the trap of a reward of zero and early stagnation in local optima. The policy agent undergoes multi-stage training, with each stage dynamically sampling experience pairs (hyperparameter configurations and their corresponding performance results) from the adaptively updated experience pool. After each training stage, the experience pool is refined by incorporating newly generated hyperparameter configurations and their performance results. Both the policy agent and reference model are updated via exponential moving average (EMA)³⁸ as a new starting point.

A key challenge in RL-based HPO is the substantial computational overhead incurred in reward evaluation, which involves full model training and validation — particularly prohibitive for large-scale datasets. This constitutes a critical bottleneck where delayed reward feedback impedes efficient policy learning. We address this bottleneck based on an important insight: during policy optimization, only the relative performance difference between hyperparameter configurations is required, while absolute performance evaluation can be deferred to final testing. Building upon the statistical principle that the distributions of data subsets correlate strongly with that of complete datasets (especially in large-scale data regimes), we propose a reward approximation mechanism, which estimates relative hyperparameter quality by training on split (or sampled) subsets of the entire dataset. This mechanism maintains the performance rank consistency while reducing the cost of reward computation by up to 80×. Full-dataset evaluation is only employed in the final evaluation.

Contributions: In this work, we introduce Rocket, a recurrent reinforcement learning framework that transforms hyperparameter optimization from a manually intensive, limited-scale procedure into a fully automated, scalable, and efficient process. The core contributions of this work are fourfold: (1) Rocket shrugs off the dependency on manually curated experience or domain-specific priors, enabling truly prior-free, mixed-type hyperparameter optimization across a wide range of tasks automatically. (2) We design a self-improving policy agent trained through a staged reinforcement learning scheme, which iteratively refines its strategy via interaction with the target model. (3) To overcome the prohibitive computational cost of evaluating large-scale datasets, we develop a reward approximation mechanism that accelerates the policy learning phase by up to 80 × while maintaining reliable guidance. (4) Extensive experiments on eight diverse tasks across 32 benchmarks demonstrate that Rocket consistently achieves state-of-the-art performance, outperforming the vast majority of expert-tuned results reported in the literature. More importantly, real-world deployment validates its practicality, yielding a 13.4 × reduction in optimization time and a 73% decrease in cost. Collectively, these advances establish Rocket as an automated, prior-free, and production-ready solution that moves hyperparameter optimization beyond constrained experimental settings into practical, large-scale utility. All the code and engineering resources will be released to the public to support further research and industrial adoption.

Results

Baseline selection

For classical HPO baselines, we provide a comparative evaluation of Rocket against two established Bayesian-based approaches: Bayesian optimization with Gaussian process (BOGP)⁹ and Bayesian optimization with hyperband (BOHB)¹³, as presented in “Characteristics of Rocket“.

For reinforcement learning (RL)-based approaches, we exclude traditional value-based RL baselines (e.g., refs. ^23,24) as they are theoretically ill-suited for this problem. They require discretizing the continuous hyperparameter space, which triggers a combinatorial explosion in high-dimensional settings (optimizing 20+ hyperparameters). Comparing Rocket against baselines that are structurally incapable of handling such continuous HPs would be scientifically unsound. Recently, LLMs have been combined with RL for further optimization, with prominent algorithms including Proximal Policy Optimization (PPO)¹⁸, Direct Preference Optimization (DPO)³⁹, and Group Relative Policy Optimization (GRPO)³⁶. Here, we also exclude PPO because it relies on training a separate value network. In HPO tasks where reward signals are sparse and expensive to obtain, training a stable value estimator is notoriously inefficient. Therefore, we focus our comparison on DPO and GRPO. The details are in “Characteristics of Rocket“. In addition, we compare our method with top-tier LLMs—including GPT-4o³⁵ and DeepSeek-R1⁴⁰, which are commonly adopted in prior work³¹. The results are presented in both “Characteristics of Rocket“ and “Performance Comparison with SOTAs”.

For SOTA comparisons, we further include the performance records from some public scoreboards such as OpenMMLab’s projects^41–43, as presented in Table 1. These results represent the culmination of extensive community tuning in the corresponding research areas. Here, we refer to them as expert-level tuning results (human experts or experts for short). Note that GPT-4o and DeepSeek-R1 are industry-standard, large-scale LLMs widely recognized for their strong performance in AutoML tasks^30,33,34. As such, they are included here as key SOTA counterparts.

Table 1.

Comparison with SOTAs of different tasks with different networks over various benchmarks

Modality	Task	Network	Benchmark	GPT-4o	DS-R1	Expert	Rocketavg	Rocketmax
Image	CLS↑	Res-1844	mini-CIFAR	69.90	61.60	73.40	77.22 ± 0.15	77.40
		Res-1844	CIFAR-100	71.26	76.60	75.61	79.56 ± 0.06	79.67
							78.54 ± 0.06	78.63†
		Res-5042	CIFAR-100	71.73	74.82	79.90	80.75 ± 0.05	80.85†
		Res-5042	CUB200	86.95	86.57	88.45	88.76 ± 0.02	88.78
		Res-1842	ImageNet	60.26	63.11	69.90	70.68 ± 0.26	71.20†
		ViT-t42	ImageNet	72.47	74.05	76.25	74.59 ± 0.50	75.34†
	DET↑	YOLOv541	VOCDET	55.00	54.00	51.52	54.98 ± 0.07	55.10
							54.72 ± 0.04	54.80†
		YOLOv541	COCO17	22.90	22.50	27.60	27.90 ± 0.09	28.00†
		FRCNN5041	COCO17	33.40	27.10	37.40	36.04 ± 0.58	37.50†
	SEG↑	HRNet-1843	VOCSEG	66.43	62.12	65.50	67.05 ± 0.71	68.27†
	OCR↑	CRNN45	CUTE80	56.94	58.33	61.46	62.57 ± 0.40	62.85†
			IIIT5K	80.40	80.73	78.50	82.49 ± 0.27	83.00†
			SVT	78.52	78.52	78.36	81.20 ± 0.32	81.76†
			SVTP	61.55	62.95	62.79	65.15 ± 0.23	65.43†
			IC13	86.50	86.40	86.21	88.79 ± 0.17	89.06†
			IC15	53.35	54.02	54.41	56.18 ± 0.28	56.52†
Text	T2V↑	CoSENT46	STS-B	77.70	77.92	78.60	78.94 ± 0.20	79.31
	TC↑	Bert47	AFQMC	69.0	73.40	74.10	74.52 ± 0.05	74.61
			TNEWS	56.78	56.59	56.93	57.25 ± 0	57.25
			IFLYTEK	59.33	59.91	59.60	61.83 ± 0	61.83
			CMNLI	79.80	79.27	80.50	80.47 ± 0.12	80.61
			OCNLI	80.05	78.52	80.39	80.82 ± 0	80.82
			WSC	63.47	74.34	75.66	77.63 ± 1.27	79.93
			CLS	67.90	50.0	81.17	81.97 ± 0	81.97
Time series	TSF↓	TTM-r248	ETTH1	0.392	0.392	0.391	0.389 ± 0	0.389
			ETTH2	0.356	0.357	0.355	0.342 ± 0.001	0.341
			ETTM1	0.380	0.379	0.380	0.376 ± 0	0.376
			ETTM2	0.283	0.280	0.284	0.268 ± 0	0.268
			Weather	0.237	0.236	0.237	0.235 ± 0	0.235
			Electricity	0.182	0.183	0.183	0.182 ± 0.001	0.181
			Traffic	0.452	0.439	0.433	0.425 ± 0	0.425
Audio	ASR↓	Whisper49	Co-Voice	54.40	36.0	32.70	31.34 ± 0.16	31.08

Rocket_avg denotes the mean performance and standard deviation across the top 10 discovered HP Configs, whereas Rocket_max reports the highest performance observed among these configurations. In practice, Rocket_max is typically deployed as the final hyperparameter configuration for deployment. “Expert” represents publicly available results by human experts. The arrow after “Task” (↑ or ↓) indicates the optimization goal for its performance metric: maximize or minimize. This standard notation clarifies whether higher or lower values are preferable for performance optimization. †means training with split data. Values in bold represent the best performance in various settings.

Performance comparison with SOTAs

To verify Rocket’s effectiveness, we conduct experiments of eight fundamental deep learning tasks — image classification (CLS), object detection (DET), semantic segmentation (SEG), optical character recognition (OCR), text embedding (T2V), text classification (TC), time series forecasting (TSF), and automatic speech recognition (ASR). These tasks cover various data modalities in deep learning, including image, text, time series, and audio. These experiments involve a total of 32 benchmarks. The network architectures, datasets, and results (including mean performance and standard deviation) are in Table 1. In practice, when the dataset is relatively small (e.g., CUB200 with 6000 images), Rocket is evaluated directly on the entire set. When the dataset is large, Rocket is operated on split (or sampled) subsets, and the final results on the full dataset are achieved with the learning rate scaling strategy. For example, the accuracy of Res-18 w. ImageNet is obtained on its split subsets, where the size of each subset is 1/128 of the entire set. Both the dataset splitting strategy and learning rate scaling strategy are detailed in “Acceleration on large training datasets” and “Accelerating rocket on large datasets”.

From Table 1, we can see that Rocket consistently outperforms human experts on almost all models. On benchmarks like CIFAR-100 and VOC_DET, Rocket outperforms human experts by substantial margins ( ~ 4.06% and  ~ 3.58%, respectively). Across all OCR evaluation datasets, Rocket consistently and significantly outperforms human experts by 1.4% to 4.5%. While gains appear modest in some cases (1%), it is important to recognize that these are classic public datasets where expert results represent the culmination of extensive community tuning, surpassing this performance ceiling in a autonomous way is a significant achievement, where even tiny gains require significant effort. That even the top LLMs like GPT-4o and DS-R1 cannot achieve this. Furthermore, the low performance standard deviation reported in Table 1 demonstrates that Rocket consistently achieves this level, confirming that the improvements are robust, rather than from random fluctuations. While gains appear modest in some cases (1%), it is important to recognize that these are classic public datasets where expert results represent the culmination of extensive community tuning, surpassing this performance ceiling in an autonomous way is a significant achievement, where even tiny gains require significant effort. That even the top LLMs like GPT-4o and DS-R1 cannot achieve this. Furthermore, the low performance standard deviation reported in Table 1 demonstrates that Rocket consistently achieves this level, confirming that the improvements are robust, rather than from random fluctuations.

Though Rocket slightly underperforms human experts on ViT-t (75.34% vs. 76.25%), the gap is marginal (<1%), indicating that Rocket has effectively achieved a utility level comparable to human experts. We attribute this slight underperformance to the use of the data splitting strategy for efficiency. Our experimental results (see Table 1) support this: ResNet-18 and YOLOv5 achieved higher performance when trained on the full dataset compared to their subset-trained counterparts (e.g., 79.67% vs. 78.62% on Res-18/CIFAR-100; 55.10% vs. 54.80% on YOLOv5/VOC_DET). This trend suggests that the slight performance gap on ViT-t stems from the information loss inherent in subset sampling, rather than the capability of Rocket itself. Therefore, it is reasonable to infer that Rocket could further fill this gap or surpass expert baselines if given the computational resources for full dataset optimization.

Here, we demonstrate conclusively that the HP Configs identified by Rocket are not sporadic results. Instead, Rocket robustly automates the attainment of SOTA on wide-range target tasks — typically surpassing almost all expert-tuned results. Beyond raw performance margins, the core value of Rocket lies in its autonomous capability to match or exceed human expertise without prior knowledge. As evidenced by our industrial deployment (see “Rocket Performance in Real Industrial Deployment“), Rocket achieves a 13.4 × reduction in optimization time and a 73% cost saving compared to manual tuning. These results show that Rocket offers a robust, cost-efficient solution for replacing labor-intensive manual workflows in real-world scenarios.

Characteristics of Rocket

Zero-experience startup capability

Rocket can launch hyperparameter optimization in a cold-start manner, without any human-designed prior knowledge.

For comparison, we use top-tier LLMs including GPT-4o³⁵ and DeepSeek-R1, as the HPO agent to iteratively receive sampled hyperparameter pairs and generate its new hyperparameters. Specifically, we first construct 10 HP Configs as the initial experience pool by LLMs with the preset prompts (See the details in Supplementary Section 3.1). Then, we obtain the input by employing the frequently-used top-2 sampling strategy on its historical experiences, i.e., selecting the top-2 hyperparameter Configs from the experience pool. The newly generated hyperparameter Configs will be verified based on the performance of the target task. Additionally, we also compare our method with two traditional HPO baselines (BOGP⁹ and BOHB¹³) as well as two popular RL-based methods: DPO³⁹ and GRPO³⁶. For each of these methods, we report the mean performance and variance across repeated runs.

Fig. 2a shows the results. We can see that it is difficult for GPT-4o to improve hyperparameters through historical experiences, and it substantially falls behind Rocket. Though DS-R1 achieves better results than GPT-4o, it still lags behind our method by a large margin. We argue that iterative HPO based on general LLMs or SFT models is not enough to solve the HPO problem. The comparison results between LLMs and Rocket in Table 1 demonstrate the superiority of Rocket. As expected, the Bayesian-based methods largely fall behind the LLM-based methods, which has also been demonstrated in ref. ³¹.

Fig. 2. Performance comparison on mini-CIFAR.

a Comparison of Rocket with mainstream large language models, reinforcement learning-based methods (DPO and GRPO), and Bayesian-based methods. Here, GPT-4o and DeepSeek-R1-0528 (DS-R1) are used as the underlying models. For Rocket, four colored line segments illustrate different stages, and “Rocket Initialization” denotes its initial performance with synthetic experiences. The vertical axis (performance) represents the average Top-1 accuracy, and the horizontal axis (Step) indicates the number of iterative steps. b Performance comparison of policy models of different sizes. Here, each point represents the average performance up to that step.

In the case of RL-based approaches, we observe that DPO surpasses GPT-4o, yet still falls behind top model DS-R1. Notably, the more advanced GRPO performs much worse than DPO. By analysis, we attribute this to GRPO’s strong reliance on cold-starting with real experiences, a phenomenon also noted in the DeepSeek-R1 paper⁴⁰. Without this cold-start process, its performance drops dramatically. In real-world scenarios, however, obtaining sufficient real experiences incurs significant human and computational costs. By contrast, our method does not rely on real-experience cold start, enabling the model to explore entirely from scratch and ultimately achieve superior performance over existing methods.

In addition, we conduct an ablation study on the size of the policy model. Specifically, we replace the base policy model with Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct, and evaluate their performance on mini-CIFAR. As shown in Fig. 2b, larger policy models (1.5B and 7B) are also capable of achieving great HPO results. Notably, larger models exhibit higher exploration efficiency in the early stages of optimization compared to the 0.5B model. However, across the full Rocket pipeline, the performance gain from scaling up to a 7B model remains marginal. Given that the 0.5B model already achieves high and competitive results, we keep it as the default policy model for all subsequent experiments.

Effect of Rocket’s stage size

Rocket’s training stages play an important role in boosting hyperparameter optimization. We first do an ablation study to evaluate the effect of stage size on mini-CIFAR (Data construction is described in “Implementation Details”). Then, we check the performance distribution of different stages on the CIFAR-100 and VOC_DET benchmarks.

In Supplementary Table 1 we present the accuracy results of different stage sizes: 1, 2, 4, 8, and 16. As the stage size increases, Rocket gradually performs better. In the subsequent experiments, we set the stage size to 16 by default. Note that when the stage size is set to 1, the training process is unstable and prone to non-convergence.

Fig. 3 shows how performance changes within each Rocket stage on CIFAR-100 and VOC_DET. The corresponding networks are ResNet18 and YOLOv5, and Top-1 accuracy and mean Average Precision (mAP) are used as their evaluation metrics. In Stage_1, most of the experiences produce poor results, leading to lower average performance. As going through the following stages, Rocket achieves more stable performance, i.e., the performance variance becomes smaller. This demonstrates the effectiveness of the multi-stage mechanism in Rocket.

Fig. 3. Performance distributions for different Rocket stages.

a and b show the performance distributions on CIFAR-100 and VOC_DET, respectively. The details of data splitting (or sampling) are described in “Acceleration on large training datasets”. Here, the vertical axis “performance” indicates Top-1 accuracy on CIFAR-100 (a) and mean average precision (mAP) on VOC_DET (b).

Hyperparameter tuning trajectory

To thoroughly explore how Rocket progressively accomplishes hyperparameter optimization, we visualize the process of hyperparameter update. First, we vectorize approximately 100,000 HP Configs generated by Rocket on mini-CIFAR, and visualize the hyperparameter space using t-SNE⁵⁰, incorporating their performance values (top-1 accuracy of classification). Then, we select the HP Configs generated by Rocket under its default setting to construct the tuning trajectory by consecutively connecting the points of hyperparameters of different steps. The constructed trajectory is illustrated in Fig. 4. Correspondingly, the changing trends of different hyperparameters are shown in Fig. 5

Fig. 4. A hyperparameter tuning trajectory in the hyperparameter space.

To construct the hyperparameter trajectory, each HP Config is represented as a feature vector, which is visualized as a point by t-SNE to a 2D space. We first choose the HP Configs generated under Rocket’s default setting. Then, for each step of each stage in Rocket’s training process, we select the median HP Config based on the task performance to represent the step and visually highlight it. Finally, we consecutively connect these highlighted points. Here, “performance” refers to Top-1 accuracy of classification on mini-CIFAR. The brighter the yellow color is, the higher the Top-1 accuracy is.

Fig. 5. The changing trends of some hyperparameters during Rocket’s tuning processing.

As in Fig. 4, the HP Configs are generated under Rocket’s default setting on mini-CIFAR. We use the median-performance HP Config per step for visualization. a shows the trends of six common HPs: “learning rate” (lr), “batch size'', “max_epochs'', “momentum'', “weight_decay'', and “num_workers''. b tracks Rocket’s optimizer choices. c shows the changing trend of parameter T_max when choosing “CosineAnnealingLR” as lr_scheduler. d plots the trends of parameters “start_factor” and “end_factor” of scheduler “LinearLR''. e visualizes the trend of parameter “gamma” and value examples of parameter “milestones” of scheduler “MultiStepLR''.

By collectively checking the results in Figs. 4, 5, we can observe the dynamic evolution process of the exploration-exploitation balance. In the early stages of RL (Stage_1 and Stage_2), the hyperparameter curves exhibit significant fluctuations, indicating that Rocket conducts extensive exploration in the hyperparameter space, attempting various hyperparameter configurations to maximize the reward. As the RL process proceeds, Rocket gradually finds the optimal ranges of certain hyperparameter configurations (e.g., “batch size” stabilizing at 32). This phase is characterized by a marked reduction in variance, signifying a strategic shift from broad search to focused exploitation. Consequently, Rocket shrinks the exploration space for converged parameters and turns more attention to refining others (such as “lr”). In the later stages of RL (Stage_3 and Stage_4), Rocket has obtained the optimal ranges of most hyperparameters, leading to focused exploration, and ultimately converges to the optimal region in the hyperparameter space.

Discovering novel HP Configs and skills without priors

Through extensive engineering practice and optimization, human experts have established a set of default hyperparameter configurations for deep learning. For instance, the “Adam” optimizer typically adopts a default learning rate of 0.001 with “betas” being set to (0.9, 0.999), while “momentum” in optimizers is often fixed at 0.9 or 0.95 to balance convergence speed and training stability. Additionally, experts have developed practical training techniques, such as the linear scaling principle (suggesting a proportional relationship between “learning rate” and “batch size”), as well as learning rate warmup and cosine annealing strategies, which have become standard practices to enhance training stability and model performance.

Although these heuristic rules provide valuable guidance, they may also constrain the exploration of novel hyperparameter configurations. Inspired by AlphaGo Zero⁵¹, which surpasses human expertise through self-play rather than relying on human game records, Rocket challenges the traditional paradigm by employing reinforcement learning to autonomously optimize hyperparameters from scratch, without leveraging human priors. We observe that Rocket can discover unconventional yet effective hyperparameter configurations and master advanced training skills:

• The “momentum” of the optimizer is not always set to 0.9 or 0.95. In different target tasks, as the other hyperparameters such as learning rate change, “momentum” should also change to adapt to the overall training process. As shown in Fig. 5a, during reinforcement learning, Rocket gradually determines that 0.3 is the optimal setting for “momentum” for this specific task.

• As shown in Supplementary Fig. 1a, without prior experiences, Rocket independently combines the “LinearLR” scheduler and “CosineAnnealingLR” scheduler to implement an effective “CosineWarmup” training strategy in engineering practice.

• When dealing with the scenarios with a small number of data samples, Rocket explores an effective cyclical learning rate adjustment strategy on its own. Instead of monotonically decreasing the learning rate, it uses multiple “CosineAnnealingLR” schedulers to make the learning rate cycle within reasonable boundary values, as shown in Supplementary Fig. 1b. This helps overcome overfitting issues in low-data scenarios. The effectiveness of the cyclical learning rate strategy has also been validated in previous methods^52,53.

These findings above demonstrate Rocket’s strong self-learning capability and reveal that well-designed reinforcement learning models can autonomously discover strategies that match or even surpass human expertise.

Acceleration on large training datasets

Generally, for small datasets, Rocket directly performs search on the whole training datasets. And for larger ones, Rocket employs data splitting to speed up reward acquisition.

That is, first dividing (by sampling) a large training dataset into multiple subsets, which constitute a list of subsets. Then, selecting one or multiple consecutive subsets (denoted as a subset span) from the list at a time for the training of Rocket.

The final performance is evaluated on the whole dataset with the learning rate scaling strategy given in “Accelerating Rocket on large datasets”.

Effect of subset size

In Supplementary Table 2 we give the data splitting results of some widely-used training datasets used in this paper. In Supplementary Table 3 we present the performance results of two tasks with the dataset splitting schemes given in Supplementary Table 2. By checking the results on CIFAR-100 and VOC_DET, we can see that even with only a small proportion of the full training dataset, the obtained hyperparameters by Rocket can still achieve good performance, even surpassing human experts (the “Expert” column in Table 1 and Supplementary Table 3). In summary, larger subset size leads to better performance, while a smaller subset size achieves competitive performance with less computational cost. Overall, unless significant performance degradation happens, we prefer to use smaller subsets for HPO.

Here, a practical issue must be addressed: namely, the minimum viable subset size for Rocket and how to reliably determine it. In principle, Rocket may fail when target models trained on a subset cannot provide reliable reward signals. That is, when performance on the subset fails to discriminate between different hyperparameter configurations. For instance, when hyperparameter optimization is performed on purely noise, the reward standard deviation (std) tends to be small and erratic; the resulting rewards reflect only noise, causing Rocket to break down. In contrast, as the subset becomes more representative of the full data, performance disparities among different hyperparameter settings become more pronounced, leading to a higher reward std and thus stronger learning signals.

Conceptually, the reward is derived from the relative performance and is normalized (see Eq. (8)). Consequently, the reward value is task-agnostic, meaning that its distribution is directly comparable across different tasks (as shown in Supplementary Fig. 2). The standard deviation of the reward directly reflects the magnitude of the advantage in reinforcement learning (refer to Equ. (13)). A reward std that is too small indicates an insufficient gradient for effective exploration, and vice versa. Therefore, the average reward standard deviation (std) can serve as an in-process monitoring signal: if it falls below a predefined threshold (0.2 by default in our experiments) during Stage-1, hyperparameter optimization on the current subset is deemed unreliable. As HPO goes, the policy model tends to converge, causing the reward std to decline. A persistent plateau or oscillation in reward std after its peak suggests that the current subset lacks sufficient signal to guide optimization. Hence, the reward std trend offers a real-time measure of subset adequacy. Specifically, if the average reward std fails to drop by at least 0.1 from its maximum by the last step, the subset should be reconsidered.

This behavior is illustrated in Supplementary Fig. 3b, where we observe that as the noise level increases (i.e., the overlap ratio decreases), the reward loses discriminative power — for overlap ratios between 0.01 and 0.1, the reward std is low and poorly differentiated, rendering the subset unrepresentative. In contrast, under low noise (overlap ≥ 0.8), the average std typically exceeds 0.2, the subset becomes highly representative, and the corresponding performance approaches SOTA level. We further observe that as the noise level decreases (i.e., the overlap ratio increases), the policy model converges more rapidly — lower noise leads to a more pronounced decline in the average reward std. At an overlap ratio of 1, for instance, the model achieves the most substantial average std reduction (a drop of 0.28) and the best final performance, underscoring the strong correlation between subset representativeness and convergence efficiency.

In addition, we aggregate the average reward std across 30 independent experiments (including those listed in Table 1), as shown in Supplementary Fig. 2. The average reward std consistently lies in [0.2, 0.6], and the observed decrease in reward std consistently exceeds 0.1 across all runs. Hence, we monitor the moving average of the reward std over the first five steps: if it remains below 0.2 throughout this early phase, the current subset is considered insufficiently representative for reliable HPO. Conversely, if it exceeds 0.2 and, as optimization proceeds (e.g., over 32 steps), the reward std decline consistently surpasses 0.1, the subset can be deemed sufficiently representative.

We further validate this observation on ImageNet by comparing the results of subset ratios of 1/128, 1/256, 1/512, and 1/1024 (see Supplementary Table 4 and Supplementary Fig. 4. At ratios of 1/1024, 1/512, and 1/256, model performance exhibits limited discriminability, and the stage-1 reward standard deviation remains small — indicating poor subset representativeness. Correspondingly, the policy model shows no clear convergence trend, with the overall decrease in reward std remaining below 0.1. Conversely, as the subset ratio increases to 1/128, we observe a monotonic improvement in both the average reward std and final performance, accompanied by a more substantial decline (0.13) in reward std throughout optimization. It is worth noting that even at the smallest ratio (1/1024), Rocket remains functional — achieving a certain level (65.23) of performance on the full dataset — but the reduced representativeness of the subsets ultimately limits the final model quality.

Consequently, we propose to use the average reward standard deviation (with a default threshold of 0.2) and its subsequent decline (with a default threshold of 0.1) as the diagnostic indicators to monitor the HPO process. In practice, we recommend configuring the subset size as large as computationally feasible within the user’s resource constraints.

Effects of subset distribution and span size

Here, we examine how subset distribution and span size affect the performance of hyperparameter optimization. We consider two data splitting strategies: imbalanced splitting and balanced splitting. In imbalanced splitting, the full dataset (CIFAR-100) is divided by sampling into 10 subsets with distinct class distributions — termed imbalanced subsets. While in balanced splitting, the full dataset is split by sampling into 10 subsets with a class distribution similar to that of the full dataset, differing only in the number of images in each class. In either case, each subset contains exactly 5000 images. Figure 6a shows the class distribution for different subsets. For simplicity, we visualize one subset of balanced splitting and three subsets of imbalanced splitting. Figure 6b shows the effect of subset span size on optimization performance. Here, “Unbalanced span size = x” indicates that span size is x with unbalanced splitting. We set x to 1, 2, 4, and 8 to fully check the effect of span size. It is interesting to note that although the subset distributions are very imbalanced (see also Fig. 6a), Rocket can still work normally. This demonstrates the robustness of Rocket and its ability to tolerate distribution differences among subsets. We also find that with the increase in span size, the overall performance shows a negligible decline. The larger the span size is, the faster it traverses the entire dataset. Therefore, it is worthwhile to sacrifice a little performance in exchange for a faster dataset traversing speed. Besides, the Rocket with balanced splitting subsets has better performance than with imbalanced subsets. In the subsequent experiments, the balanced splitting is employed, and the span size is set to 8 by default.

Fig. 6. Effects of subset distribution and span size on optimization performance.

a shows the class distributions of balanced/imbalanced subsets of CIFAR-100, where one subset is balancedly sampled (denoted by “Balanced Split'') and the remaining three subsets are unbalancedly sampled (denoted by “Unbalanced Split_1'', “Unbalanced Split_2” and “Unbalanced Split_3''). The horizontal axis represents the indices of the categories of CIFAR-100. The vertical axis represents the image ratio of each category. b shows the results using imbalanced subsets with span size of 1, 2, 4 and 8. Besides, the results using balanced subsets with span size = 8 are also illustrated. Here, the vertical axis indicates the top-1 accuracy on the full dataset.

A natural follow-up question is: at what level of imbalance does this strategy break down? Essentially, failure occurs when the model trained on a subset cannot obtain meaningful reward signals from the validation set. Consider an extreme scenario in the classification task, if the training subset contains only classes {A, B, C} while the validation set consists of completely different classes {D, E, F}, i.e., the training and validation sets are completely or nearly orthogonal in class space, our approach will likely fail. To illustrate this boundary, we conducted additional experiments where the class overlap ratio between the training subset and the validation set is varied from 0.01 to 1, as shown in Supplementary Fig. 3a. We observe that when the overlap ratio is low (e.g., 0.01), the HP Configs discovered by Rocket tend to yield lower performance (59.9%) with high performance standard deviation (≈15%), indicating their unreliability under such conditions. In contrast, with higher overlap ratios (e.g., 0.8), the discovered configurations exhibit much higher stability, achieving higher average performance (77.6%) with lower standard deviation (0.36%). When the ratio approaches 1, the best performance (78.3%) and a low standard deviation (0.28%) are observed. Notably, performance variance shows a generally decreasing trend with increasing ratio. In practice, such an extreme split (e.g., overlap ratio = 0.8) is impossible. Moreover, our method of constructing subsets via random sampling inherently mitigates the effect of class-level orthogonality. These results demonstrate the robustness and stability of Rocket even under extreme unbalanced conditions.

From another perspective, a lower overlap ratio corresponds to noisier and less representative subsets, and vice versa, as shown in Supplementary Fig. 3b. When the overlap ratio is small (e.g., ≤ 0.5), the resulting reward standard deviation is also small, constraining Rocket’s explorable space (evidenced by a limited decline in average reward std). In contrast, under low-noise conditions with an overlap of 0.8 — where the average reward std exceeds 0.2 — the final performance is significantly better. In addition, a clear trend emerges as the overlap ratio increases from 0.01 to 1. The average reward decrease amount rises steadily, going from 0 to 0.06, then to 0.06, 0.13, 0.14, and finally reaching 0.28, while the policy model converges at an increasingly faster pace. This pattern reinforces the strong correlation between subset representativeness and convergence efficiency, which confirms the analysis of subset representativeness in “Acceleration on large training datasets”.

Performance correlation between the subset and the full dataset

As we mentioned in “Introduction”, the distributions of subsets are positively correlated with the distribution of the whole set when random sampling is employed for dataset splitting, especially on large datasets. In addition to the learning rate, the hyperparameters searched on subsets should be consistent with those on the full dataset. Figure 7 shows the correlation between subset performance and fullset performance with the square-root learning rate scaling strategy (“Accelerating Rocket on large datasets”). We find that the performance curve with learning rate scaling (using subsets) maintains a high positive correlation without scaling (using full set), especially in the high subset performance regions. As the subset performance improves, the growth of final performance on the entire set tends to be saturated. This suggests that the hyperparameter configurations searched by Rocket on subsets generalize well, and the learning rate scaling strategy is effective.

Fig. 7. Correlation between subset and full-set performance.

a and b show the Top-1 accuracy results on CIFAR-100 with ResNet-18 and the mAP results on VOC_DET with YOLOv5, respectively. The horizontal axis represents the subset performance, while the vertical axis represents the full-set performance.

Comparison of square-root and linear scaling strategy

As described in “Accelerating Rocket on large datasets”, linear scaling is another widely adopted strategy alongside square-root scaling. To evaluate these two schemes, we conduct scaling validation on two large-scale datasets: COCO (118k images) and ImageNet (1.28 million images). The training curves for different target tasks are shown in Supplementary Fig. 5. Our results show that square root scaling yields superior performance. This advantage stems from the fact that the learning rate under square root scaling is initially larger than that under linear scaling. Starting optimization with a larger learning rate provides stronger learning momentum and helps the model avoid premature convergence to suboptimal solutions.

Effects of reward terms

We evaluate the effects of reward terms in Eqs. (8)–(10) by checking how different combinations of the three terms affect the performance on the mini-CIFAR dataset. The experimental configuration is as follows: 16 steps per stage across a total of four stages, with both the group and batch sizes fixed at 8.

Table 2 presents the performance results of different combinations of the three reward terms r_g, r_dir and r_dif (detailed in “Reward function“) by different values of coefficients β and η on mini-CIFAR with ResNet-18. We can see that both r_dir and r_dif can help to improve performance in terms of average accuracy (avg-acc) and best accuracy (best-acc). And for either of them, Rocket works well when the tunable parameter is set to 0.25. Supplementary Fig. 6 shows how the reward value and the corresponding performance change under these two configurations during the learning process. It is worth noting that r_g should be regarded as the base reward, while r_dir and r_dif are used as auxiliary reward terms for Rocket training. Further analysis on the effects of r_dir and r_dif are as follows.

Table 2.

Performance comparison of reward term combinations on mini-CIFAR

rg	β	η	Stage1		Stage2		Stage3		Stage4		Overall
			avg-acc↑	best-acc↑	avg-acc↑	best-acc↑	avg-acc↑	best-acc↑	avg-acc↑	best-acc↑	best-acc↑
✓			55.45	76.20	68.26	75.80	73.57	75.50	75.14	75.30	76.20
✓ ✓ ✓	0.05 0.25 1.0		51.52 57.74 57.32	61.60 75.30 75.60	71.11 70.51 71.02	76.40 77.10 76.10	74.30 74.57 74.18	77.30 77.30 76.30	74.79 75.00 74.45	77.00 76.80 77.00	77.30 77.30 77.00
✓ ✓ ✓		0.05 0.25 1.0	56.6 58.11 49.04	74.80 75.60 74.60	70.89 67.79 62.50	76.70 76.60 75.90	73.98 73.94 62.84	76.30 77.10 76.60	75.98 74.55 72.59	77.00 77.30 77.10	77.00 77.30 77.10
combining alternating relaying			53.16 51.61 57.56	74.20 75.30 75.60	68.51 64.96 73.01	75.40 75.00 77.40	72.93 70.56 74.70	77.10 76.00 77.40	73.32 73.48 76.12	76.90 76.90 77.10	77.10 76.90 77.40

This table presents the Top-1 accuracy results for various combinations of reward terms (r_g, r_dir, and r_dif) under different coefficients β and η using ResNet-18. {avg-acc} and {best-acc} denote the average and best accuracy within each Rocket stage. The strategies “combining”, “alternating”, and “relaying” refer to specific reward configurations across stages: 0.25r_dir + 0.25r_dif on all stages, [0.25r_dir, 0.25r_dif, 0.25r_dir, 0.25r_dif] per stage, and [0.25r_dir, 0.25r_dif, 0.25r_dif, 0.25r_dif] per stage, respectively. Bold values indicate the best performance for each setting.

Term r_dir functions as a momentum-based acceleration mechanism. By aligning the update direction with historical improvement trajectories via cosine similarity and scaling based on performance sensitivity, it maintains inertia along effective search paths. This encourages the exploitation of superior historical experiences, accelerating the convergence of Rocket toward high-performance HP configs. As training goes, this reward term increases, reflecting its role in promoting convergence (see Supplementary Fig. 6a). Consequently, it accelerates the target model to achieve optimal performance. As shown in Table 2, incorporating r_dir (vs. the baseline reward r_g) enables Rocket to achieve higher avg-acc and best-acc in Stage 2. Optimal results are achieved when the coefficient β is set to 0.25.

Term r_dif serves as a dynamic entropy regularization mechanism. It mitigates mode collapse — a phenomenon that the agent generates repetitive configurations, and local optima entrapment by penalizing similarity to previous states. This forces the policy distribution to maintain high variance, thereby differentiating hyperparameter exploration, such as the gradually decreasing reward values in Supplementary Fig. 6b. This improves peak performance, yielding higher best-acc in Stages 2-4, but reduces avg-acc, as evidenced by the performance change under r_g + ηr_dif for different η values in Table 2. Setting η = 0.25 balances avg-acc and best-acc effectively.

We also check three other different combinations of r_dir and r_dif, and find that the replaying strategy ([0.25r_dir, 0.25r_dif, 0.25r_dif, 0.25r_dif] on each stage) achieves better avg-acc and best-acc results than the combining strategy (0.25r_dir + 0.25r_dif for each stage) and the alternating strategy ([0.25r_dir, 0.25r_dif, 0.25r_dir, 0.25r_dif] on each stage). The replaying strategy is used by default in the other experiments.

Computational cost

Rocket employs reinforcement learning for hyperparameter optimization, delivering high performance at the expense of increased computational cost. This cost arises from two sources: (1) training the policy model to generate hyperparameter configurations, and (2) evaluating the target task to obtain reward signals. The overall computational cost can be evaluated as:

$$\mathrm{Total\; Cost}={\sum }_1^K{\sum }_{i=1}^{B\times N}\left(C_p+C_t\times \mid X\mid \times E_i\right)$$ 1

Here, C_p denotes the computational cost per iteration of the policy model, K is the number of reinforcement learning steps, while B and N represent the batch size and group size in Rocket, respectively. C_t refers to the per-sample computational cost of the target task, ∣X∣ is the size of the target training dataset, and E_i is the number of corresponding training epochs. The terms C_p and C_t are typically fixed. Note that B × N operations are computed in parallel, reflecting the GPU memory footprint, whereas K × B × N collectively represents the total number of trials during reinforcement learning. Optimizing K therefore directly reduces the overall computational overhead (as discussed in “Discussion”). Since the total computational cost is essentially determined, the number of available GPU resources primarily affects the user-facing latency, rather than the total computational requirement.

For the policy model, its hardware specifications are detailed in Supplementary Table 5. Here, a larger B typically demands more GPU memory. Note that reducing the group size N could also reduce memory usage, but it is not recommended as it will undermine the reliability of the relative advantage estimates. As presented in Supplementary Table 5, training the 0.5B policy models requires only a single GPU, for which the recommended GPU types (e.g., RTX4070, RTX4090, or L40) are also listed. We can see that the policy model’s optimization cost is low and controllable, it is not Rocket’s bottleneck.

The predominant computational expense stems from the repeated training of the target model to generate rewards for the policy agent — a cost that scales with dataset size and model complexity, corresponding to the term K × B × N × (C_t × ∣X∣ × E). For the target task, the GPU memory requirement is determined by the model size, batch size, and input size (e.g., ResNet-18 with a batch size of 64 and image size of 32 × 32 requires approximately 0.08GB of memory). The number of GPUs needed depends on the user’s tolerance for reward-signal latency: more GPUs linearly reduce the waiting time. For illustration, training 50,000 images with ResNet-18 at a batch size of 64 and a maximum epoch of 200 on an L40 GPU takes about 273.37 seconds. Without early stopping, if B × N = 64 and only one L40 is available, each Rocket step would require approximately 4.86 hours; with 32 L40 GPUs, the time per Rocket’s step reduces proportionally to approximately 9.11 minutes. To reduce the overhead, especially for large datasets or models, we propose the data split strategy (Acceleration on large training datasets) to accelerate training and reduce resource cost. As visualized in Supplementary Fig. 7, data split significantly reduces Rocket’s computational cost on various tasks. For example, the cost of training ResNet-18 on ImageNet is about 9.7 × 10² EFLOPs without data split, but only 1.2 × 10¹ EFLOPs with data split — an approximately 80-fold reduction. In our implementation, Rocket can be executed on a single GPU via time-division multiplexing. However, this introduces additional overhead due to alternating between policy and target model computations. We therefore recommend using a minimum of two GPUs for optimal efficiency: one dedicated to the policy model and another for the target task, thereby eliminating task-switching latency.

In practice, since the industry typically possesses substantial computational resources, Rocket’s requirements can be readily met when sufficient GPUs are available.

It is also worthy of pointing out that HPO is essentially a “once and for all” investment — once a model is optimized, it can be deployed and used until a re-optimization cycle is launched. From this perspective, Rocket is a practical and cost-effective solution for real-world applications.

Rocket performance in real industrial deployment

EZVIZ is a Hangzhou-based listed company specialized in smart home appliances, with over 500 different stock-keeping units (SKUs). Different SKUs are often equipped with different chip platforms, so engineers need to tailor and adapt specific neural network models and then optimize them. Throughout the entire process, engineers have to familiarize themselves with the specific network structures each time, set relevant hyperparameters, and conduct debugging and verification, which consumes a huge amount of engineering manpower. Moreover, when the evaluation indicators change or the model “loss” is redesigned, further efforts are needed to verify the effectiveness. This accounts for more than 35% of the workload of algorithm engineers, while suboptimal configurations contribute to approximately 29% of project delays in practice. Take a real case, for example, an engineer trained a classification model with incompatible hyperparameters: 3000 epochs, an initial learning rate of 0.1, and a step decay reducing the rate by a factor of 10 every 150 epochs. This means the learning rate will be reduced by a factor of 10⁻²⁰. Notably, such misconfigurations not only waste computational resources but also amplify development risks in time-sensitive industrial workflows.

Rocket has been deployed in EZVIZ’s AI division for several months. Supplementary Table 6 presents some preliminary performance results. In our practice, Rocket largely accelerates the optimization of deep learning model cycles by 13.4 × compared to manual methods (9 days per task) while reducing human configuration errors. In the vast majority of cases, Rocket performance is superior to that of human-trained models, i.e., 93% vs. 7%. Although Rocket underperforms humans in 7% of deep learning tasks, the performance gap was marginal, falling within a 1% difference. From an economic perspective, Rocket only costs 463.5$, significantly lower than the cost of manual development (1709.3$). These results validate Rocket’s capacity to mitigate the suboptimal problem and enhance productivity in industrial AI workflows.

Note that the AI model development cost reported here refers specifically to the cost of the model training and optimization, excluding any expense related to system integration. To further illustrate the comparative effectiveness of Rocket against manual expert tuning, we present one real-world case study: a face-liveness identification task. As shown in Supplementary Fig. 8, Rocket easily surpasses manual hyperparameter tuning, both in terms of convergence behavior during training and the best performance. In Supplementary Fig. 8, we also list the specific hyperparameter settings chosen by human expert alongside those discovered by Rocket. Our experiments reveal that Rocket selects a learning rate two orders of magnitude smaller than the human expert’s choice (0.1 vs. 0.00078818), yet pairs it with a weight-decay value an order of magnitude larger (5 × 10⁻⁵ vs. 4.7444 × 10⁻⁴) — a counterintuitive combination unlikely to be conceived via manual tuning. Furthermore, Rocket tends to identify irregular, non-standard values (e.g., an irregular momentum value of 0.923123742), whereas human-tuned parameters often cluster on rounded, grid-like points. We argue that human biases, such as the preference for round number,s are not necessarily optimal for deep learning systems, as they may not align with the true response surface of the model.

Currently, Rocket has been applied to over ten types of deep learning tasks, including object detection (with more than 50 specific tasks), object classification (with more than 20 specific tasks), speech recognition (with more than 5 specific tasks), OCR (with more than 30 specific tasks), and AI voice assistants etc.

Discussion

In this study, we addressed automated hyperparameter optimization for deep learning models. Equipped with reinforcement learning, we proposed the Rocket method that demonstrates superior performance and robustness compared to prior methods in extensive experiments across eight different learning tasks on 32 benchmark datasets of four distinct modalities (images, text, time series, and audio). Rocket also exhibits above “expert-level” performance in the vast majority of scenarios.

Rocket employs recurrent reinforcement learning with experience replay, enabling the policy agent to rapidly converge toward optimal hyperparameter ranges. To this end, we have effectively solved three key technical issues:

• Reward design. The reward function of Rocket comprises three components: a primary reward driven by the performance of the target model, and two auxiliary rewards (r_dir and r_dif) designed based on relationships between hyperparameters. These auxiliary rewards balance the exploitation and exploration of hyperparameter experiences. Specifically, r_dir facilitates rapid convergence of the policy agent toward a locally optimal region, accelerating training and enabling early stopping. Conversely, r_dif deliberately enhances the discovery of superior hyperparameter configurations by differentiating the parameter search. Consequently, we initially apply r_dir during early training rounds, subsequently leveraging r_dif to refine hyperparameters within the identified promising search region.

• Efficiently training models on large-scale datasets. We proposed an effective data splitting strategy that significantly reduces computational cost. Here, a critical issue is the minimum viable subset size required to maintain validity. Since we used the relative performance as the reward signal, the strategy may fail if hyperparameter configurations yield identical performance across all the subsets. That is, it remains effective unless model performance is indistinguishable under different hyperparameters on any subset. Practically, given fixed computational resources and acceptable latency requirements, we just make the subset size large enough to preserve distributional alignment with the full dataset.

• Determining the learning stage size. Intuitively, the Rocket’s stage size should be neither too large nor too small. An excessively small size (e.g., equal to 1) risks policy agent collapse and impedes achieving optimal performance, while an overly large size will lead to better performance but incurs high computational cost. Therefore, we maximize the stage size up to the constraint of available computational resources.

Rocket’s strength lies in its ability to achieve high-performance tuning without prior domain expertise. Crucially, it generalizes across diverse deep learning tasks, dynamically adapting to varying task structures, optimization landscapes, and performance metrics. This generalization capability eliminates the need for task-specific search space engineering, broadening the framework’s application in real-world scenarios — such as machinery fault diagnosis — where scaling manual tuning expertise across applications is impractical. By automating the most expertise-intensive aspects of model optimization, our method simultaneously lowers the barrier of entry to deep learning — enabling engineers with limited expertise to achieve state-of-the-art results — and frees senior practitioners to pursue innovative tasks like developing cross-modal learning strategies. Consequently, Rocket transforms the role of algorithm engineers from intensive manual hyperparameter tuning to high-level system design and problem formulation. This is leading to a new paradigm of human-AI collaboration where engineers specify “what to optimize” while our system autonomously focuses on “how to optimize”, potentially reducing iteration cycles in industrial applications by orders of magnitude.

However, there are still some limitations with Rocket. For example,

• Weak optimality guarantee. Rocket may identify high-performing yet potentially suboptimal hyperparameter configurations due to (1) a high-dimensional search space — with a search complexity of ${\mathbb{R}}^D$ where D is the number of hyperparameters — precludes a global optimality guarantee. (2) The performance-related hyperparameter response surface frequently exhibits non-convex, multimodal characteristics, which invalidate the guarantee of unique optima. From a reinforcement learning perspective, state transition probabilities remain unknown in such model-free approaches, preventing precise environmental modeling. To address this issue, on the one hand, we can amplify the coefficient of r_dif to suppress the convergence of the policy agent to local optima and enhance the model’s exploration capability. On the other hand, drawing on the Upper Confidence Bound (UCB) method⁵⁴, we estimate the uncertainty of the policy agent’s actions (i.e., HP configs) — where uncertainty is quantified by metrics such as action visit counts or hyperparameter similarity. Actions with higher uncertainty are assigned greater weights, prioritizing the exploration of “potentially optimal but insufficiently verified” actions.

• High computational intensity. Rocket incurs substantial computational demands for hyperparameter optimization of very large-scale model training. For example, when hardware resources are constrained ( < 100 A800 GPUs), validating massive models (e.g., >1B parameters) on large-scale datasets (e.g., >1M samples) becomes currently infeasible — though scaling to exascale compute clusters may enable such scenarios. In practice, some mitigation strategies can be applied. For example, (1) early stopping the policy agent to shorten the reinforcement learning process by terminating unpromising steps early (typically 2 × speedup). (2) Introducing additional human expertise (e.g., training records from similar target tasks) to provide Rocket with a better initialization starting point, thus achieving faster convergence.

Another concern may be the optimization of Rocket’s own hyperparameters (e.g., β and η). While applying Rocket to optimize its own hyperparameters is conceptually straightforward, it would incur a quadratic increase in computational cost. In fact, this falls under the concept of AutoRL⁵⁵ and is highly challenging. Fortunately, Rocket itself involves only several key hyperparameters, making grid search — a commonly used and effective HPO method — suitable for tuning Rocket’s own configuration. To validate this design, we perform a stability ablation study on Rocket’s key hyperparameters using a grid-search-like approach. As reported in “Characteristics of rocket”, performance is stable when the stage size exceeds 4. Furthermore, “Effects of reward terms“ shows that incorporating the terms r_dir and r_dif consistently improves results, with optimal values observed around β = η = 0.25. These findings confirm that Rocket is robust to its core hyperparameters. Importantly, in various target tasks, all experiments use the same internal hyperparameter set for Rocket, no task-specific tuning of stage size, β, or η is required. This demonstrates the general applicability of Rocket.

In summary, Rocket has demonstrated its substantial potential for advancing hyperparameter optimization in deep learning processes, along with significant industrial viability. To further strengthen its performance and broaden its application, future works will focus on (1) constraining the search direction within the reinforcement learning framework to boost optimization efficiency, potentially through structured action space pruning. (2) Deploying Rocket to more real scenarios to evaluate its performance and expose its limitations, with which to guide its improvement and update.

Methods

Overview

The working mechanism of Rocket is illustrated in Fig. 8a. We define F_θ as the target neural network (or model) with trainable weights θ, and π_ω is the policy agent with learnable weights ω. I and X are the format prompts of the policy agent and input data for training F_θ, respectively. The hyperparameters of F_θ are denoted as ϕ = {φ₁, φ₂,...}, and the corresponding performance results of F_θ for different tasks are G = {g₁, g₂,...}. For example, g_i can be the top-1 accuracy for the classification task and MAP-50 for the detection task.

Fig. 8. Overview of the Rocket framework and its optimization mechanisms.

a The overall working mechanism of Rocket. The policy agent π_ω is recurrently trained with group reinforcement learning, while the reference model π_ref is pre-trained on a cold-start dataset. The superscript t denotes the t-th stage in the iterative training process. Both the experience pool and the reference model are dynamically updated during training to enable adaptive optimization. The symbol ≈ indicates that approximate rewards are used, which are obtained via a data-splitting strategy for large datasets. b The recurrent policy optimization process initialized with cold-start experience, where K steps of policy learning are performed at each stage. c Illustration of the acceleration mechanism on large datasets. The dataset is divided into equal-size subsets [...,Xi,...], and a subset span of size s (here s = 8) is sequentially and recurrently selected for reward computation.

We train the policy agent π_ω by the reinforcement learning strategy with respect to (X, ϕ, G, F_θ). In Fig. 8a, Φ is the sampling strategy for obtaining experience pairs from the experience pool E^t as input. The reference model π_ref is the constraint part of reinforcement learning. The superscript lowercase t means the t-th stage in the iterative training process.

Recurrent HPO paradigm

Given the target task and corresponding training data, the learned weights θ of F_θ are uniquely determined by its hyperparameters (e.g., optimizer choice, learning rate). Inspired by human engineers who iteratively refine hyperparameters through trial and error, drawing on historical experience to guide new exploration toward better configurations, we propose to use a policy agent π_ω that generates proper hyperparameters by leveraging past experiences — specifically, the history of hyperparameters (ϕ) and their corresponding performance results (G). As a result, the performance of F_θ is effectively governed by the policy agent π_ω and its accumulated experiences (ϕ, G). Details of the Rocket process are given in Supplementary Section 5.

Cold-start experience initialization

To initialize Rocket, we construct an initial experience pool E⁰, which is also called the cold-start dataset. This dataset is sampled from the predefined JSON schema with randomized parameter values. This synthetic generation serves strictly as a syntactic constraint to ensure valid interaction by teaching the agent the correct JSON output format, rather than imparting any domain knowledge. Since the values are selected randomly within the valid ranges, they represent an uninformative prior that contains no information about the correlation between hyperparameters and performance. This design ensures that while the agent learns the communication protocol, the actual exploration of the hyperparameter landscape starts from a strictly prior-free state. The detailed process is described in Supplementary Section 3.3.

During the iterative training process, the experience pool E^t of stage t undergoes dynamic refinement through an evolutionary mechanism: at stage t, the policy agent π_ω generates new hyperparameter configurations, which are systematically incorporated into E^t+1 of stage (t+1). Concurrently, the experience sampling strategy Φ produces an optimized training sequence $\left[{\mathcal{D}}^0,{\mathcal{D}}^1,...,{\mathcal{D}}^T\right]$, where ${\mathcal{D}}^t$ indicates the current state of hyperparameter knowledge, which is also called query set.

Sampling strategy Φ

In order to train the policy agent π_ω, we retrieve the input pairs from historic experience pool E by the sampling strategy Φ, that is,

$$\left({\varphi }^{*},G^{*}\right)=\mathcal{D}={\mathrm{\Phi }}_{\mathrm{K}}\left(E\right)={\mathrm{\Phi }}_{\mathrm{K}}\left({\varphi }_{\left[1:\mid E\mid \right]},G_{\left[1:\mid E\mid \right]}\right)=\left\{...,\left[\left({\phi }_i,g_i\right),\left({\phi }_j,g_j\right)\right],...\right\}s.t.i\ne j\mathrm{and}{g}_i\ne g_j.$$ 2

Above, Φ_K means to sample K candidate pairs of hyperparameter configurations and their associated performance values. {⋅,⋅} is the binary tuple of (hyperparameter, performance). [⋅,⋅] stands for the input pairs consisting of different combinations of hyperparameter configurations and their corresponding performance values.

However, in policy learning, we observe that experience sampling in a pure top-performing mode (commonly used in previous methods³¹) will lead to premature convergence. This occurs because it inherently limits the diversity of sampled experiences, thereby constraining the exploratory capacity of the reinforcement learning agent. To overcome this exploration bottleneck while maintaining training stability, we implement Φ with a randomized sampling strategy to ensure (1) comprehensive coverage of the large hyperparameter search space, and (2) non-greedy exploration to avoid local optima.

Reference model

The reference model is a common component in reinforcement learning, serving two primary functions: preventing policy degeneration — where short-term reward over-optimization leads to degraded output quality, and constraining update magnitude via KL divergence regularization to maintain the reliability and stability of the generated content. By providing structured prior knowledge about desirable behaviors, it acts as a critical anchor for safe and sample-efficient decision-making. In our work, the reference model and the policy model share the same architecture Qwen2.5-0.5B-Instruct.

Rocket adopts a dynamic reference model framework, where the reference model is iteratively synchronized with the policy model from the preceding round. Formally, the initialization rule of the reference model is: ${\pi }_{ref}^{t+1}\leftarrow {\pi }_{\omega }^t$, where ${\pi }_{\omega }^t$ denotes the t-th policy model at the end of round t after all parameter updates. Both ${\pi }_{\omega }^0$ and ${\pi }_{ref}^0$ are implemented by LoRA-based SFT⁵⁶ of Qwen2.5-0.5B-Instruct.

Notably, the model’s initial performance is intentionally unconstrained. That is, the only requirement is syntactic correctness in hyperparameter generation, ensuring the output conforms to the prescribed JSON schema. More details are given in Supplementary Section 3.3.

Recurrent policy update

A LLM-based model is used as the policy agent π, and each generated HP Config φ_i ∈ ϕ^* is regarded as an action. While the target network F_θ can be treated as the reward function, and each performance value g_i ∈ G^* is regarded as a reward. Correspondingly, experiences sampled by Φ can be regarded as a query set ${\mathcal{D}}^t\in D^T$. Therefore, reinforcement learning can be applied to solving the policy optimization problem.

As the training process goes, the existing data distribution is difficult to provide additional experience information. However, reinforcement learning can generate additional experience records, so we can further expand the exploration space and experience utilization by updating the experience pool and the corresponding reference model. Thus, we optimize the policy agent recurrently with group policy optimization (abbr. RGPO) as in Ref. ³⁶.

Concretely, we implement a recurrent policy optimization mechanism where the trained policy agent π_ω dynamically produces HP Configs conditioned on historical experience tuples (ϕ, G). This process establishes a closed-loop optimization system that is divided into three parts as follows:

HP Config generation. At stage t, the policy agent generates HP hyperparameter configurations, i.e.,

$${\varphi }^t={\pi }_{\omega }^t\left(I,{\varphi }^{*},G^{*}\right).$$ 3

Performance-reward coupling. The result G^t serves as both the target network’s performance measure and the reinforcement learning reward signal, that is,

$$G^t=F_{\theta }\left(X,{\varphi }^t\right).$$ 4

Recurrent update rule. The policy undergoes incremental refinement via

$${\pi }_{\omega }^{t+1}=\mathrm{Rocket}\left({\pi }_{\omega }^t,E^t\right),$$ 5

$$E^{t+1}=E^t\cup \left({\varphi }^t,G^t\right).$$ 6

Objective function

Formally, given a query set ${\mathcal{D}}^t$, the objective function of Rocket is as follows:

$${\mathcal{J}}_{RGPO}^t\left(\omega \right)=\mathbb{E}\left[q~{\mathcal{D}}^t,{\left\{{\phi }_i\right\}}_{i=1}^N~{\pi }_{{\omega }_{old}}^t\left({\varphi }^t\mid q\right)\right]\cdot E{A}^t.$$ 7

Here, ${\pi }_{{\omega }_{old}}^t$ is the old policy agent, and EA is the overall estimated advantage whose calculation is detailed in “Recurrent group policy optimization”.

Hyperparameter search space

To validate the autonomous optimization capability of Rocket that does not rely on sensitive hand-crafted priors, we establish a broad and generalized hyperparameter search space. The construction of this space follows two key principles. First, regarding search dimensionality and selection, instead of picking specific parameters, our selection is strictly grounded on the standard adjustable configurations available in mainstream open-source frameworks such as MMPretrain⁴² and MMDetection⁴¹. Consequently, Rocket simultaneously optimizes over 20 concurrent hyperparameters, including optimizers, learning rate schedulers, and regularization terms. This dimensionality significantly exceeds that of prior RL-based methods^23,24 that typically focus on a small subset of discrete parameters. Second, regarding parameter ranges, we adopt broad and continuous boundaries. For example, setting the learning rate within the interval (0, 1), which is delineated by domain expertise to encompass virtually the entire spectrum of plausible tuning choices. Unlike limited discrete grids used in prior works, this creates a search complexity with theoretically infinite choices. Rocket’s performance under these unconstrained conditions demonstrates its robustness, indicating that its success is driven by the policy agent’s learning capability rather than sensitive search space engineering. Detailed definitions of the hyperparameter space are provided in Supplementary Section 4.

Recurrent group policy learning

Reward function

Different from classical reinforcement processes, we do not need a pre-trained model as the reward model. Because the target network F_θ can output the scalar performance value w.r.t the output of the policy agent, hence we use the sampled experience ${\left({\varphi }^{*},G^{*}\right)}_k$ as the input for the k-th step at stage t, and both F_θ and the last experience pool E^t−1 provide the rewards for the k-th reinforcement step. Formally, the input pairs of π_ω are denoted as $\left[\left(\underset{\_}{\phi },\underset{\_}{g}\right),\left(\overline{\phi },ḡ\right)\right]$, and the output is denoted as $\left(\stackrel{°}{\phi },\stackrel{°}{g}\right)$. Here, $\left(\overline{\phi },ḡ\right)$ means the hyperparameter-performance pair with higher performance in the inputs, while $\left(\underset{\_}{\phi },\underset{\_}{g}\right)$ is the opposite. We also use the historical experience with the highest performance $\left(\widehat{\varphi },\widehat{g}\right)$ as a reference.

We first consider the main reward term inferred from performance. We use the performance increment ($\stackrel{°}{g}-ḡ$) to motivate performance improvement as follows:

$$r_g=\min \left(-1+e^{\frac{\stackrel{°}{g}-ḡ}{\widehat{g}-ḡ+ϵ}},e-1\right)$$ 8

If $\stackrel{°}{g}$ exceeds $ḡ$, the reward is positive, and the larger the difference between $\stackrel{°}{g}$ and $ḡ$ is, the bigger the reward is. Otherwise, it will be punished. In addition, we should remember that the higher $ḡ$ is, the harder it is to generate a new $\stackrel{°}{\phi }$ whose performance $\stackrel{°}{g}$ is larger than $ḡ$. This is because the performance of the target model is always limited to a certain extent; finding better hyperparameters is always more challenging. Hence, we use the difference between $\widehat{g}$ and $ḡ$ to amplify the reward. Distinctively, if $\stackrel{°}{g}=\widehat{g}$, the reward is close to e−1. While $\stackrel{°}{g}>\widehat{g}$, we just update $\widehat{g}$, and the reward is set to e−1. Here, ϵ is used to prevent division by zero, which is set to 1e-8 by default. e is the natural constant.

To enhance the exploitation of superior historical hyperparameters in policy learning, we consider another reward based on the cosine similarity between $\stackrel{°}{\phi }-\underset{\_}{\phi }$ and $\overline{\phi }-\underset{\_}{\phi }$. This mechanism functions analogously to momentum-based acceleration in gradient descent. The cosine similarity term determines the update direction by aligning it with historical improvement trajectories, ensuring the agent maintains inertia along the effective search paths. Meanwhile, the ratio of $\mathrm{\Delta }g=\mid \mid ḡ-\underset{\_}{g}\mid \mid$ to $\mathrm{\Delta }\phi =\mid \mid 1-\cos \left(\overline{\phi },\underset{\_}{\phi }\right)\mid \mid$ serves as a sensitivity scaling factor. This promotes the search toward directions where minor hyperparameter changes yield significant performance gains. We call it directional reward, denoted by r_dir. Formally, r_dir is evaluated by

$$r_{dir}=\beta \cdot \mathrm{clip}\left(\frac{\mathrm{\Delta }g}{\mathrm{\Delta }\phi +ϵ}\cdot \cos \left(\stackrel{°}{\phi }-\underset{\_}{\phi },\overline{\phi }-\underset{\_}{\phi }\right),-\frac{e-1}{\beta },\frac{e-1}{\beta }\right)$$ 9

where β is a pre-set coefficient. The reward value is truncated between 1 − e and e − 1.

To avoid the trap of a reward of zero and local optima in Eq. (8) where $\stackrel{°}{\phi }==\overline{\phi }$ produces $\stackrel{°}{g}==ḡ$, we provide extra punishment $e^{\frac{1}{1+\mid \mid 1-\cos \left(\stackrel{°}{\phi },\overline{\phi }\right)\mid \mid }}$. Furthermore, note that the policy agent may degenerate into generating repetitive hyperparameter configurations $\widehat{\varphi }$ with identical performance $\widehat{g}$. This phenomenon is known as mode collapse, which causes the agent to be lazy in searching for better hyperparameters than $\widehat{\varphi }$. Therefore, extra punishment is needed. To promote novel hyperparameter exploration, we introduce the third reward, denoted by r_dif. This acts as a dynamic entropy regularization mechanism. By penalizing similarity to previous states, it forces the policy distribution to maintain high variance and explicitly differentiates hyperparameter exploration to escape local stagnation. It is formulated as follows:

$$r_{dif}=-\eta \cdot \left(-2e^{\frac{1}{2}}+e^{\frac{1}{1+\mid \mid 1-\cos \left(\stackrel{°}{\phi },\overline{\phi }\right)\mid \mid }}+\frac{1}{\mid \widehat{\varphi }\mid }{\sum }_{\widehat{\phi }\in \widehat{\varphi }}{e}^{\frac{1}{1+\mid \mid 1-\cos \left(\stackrel{°}{\phi },\widehat{\phi }\right)\mid \mid }}\right),$$ 10

where η is a pre-set coefficient. The overall reward is thus evaluated as follows:

$$r=r_g◇r_{dir}◇r_{dif}$$ 11

where ◇ represents the combination scheme of different reward terms, as given in “Effects of reward terms”.

Recurrent group policy optimization

The policy agent can be optimized with the cold-start experiences or cold-start dataset E⁰ (synthetic experiences) on the target task. Fig. 8b illustrates the process. Stage 0 with synthetic experiences generates corresponding real-world experiences E¹. Following Eq. (5), the (t+1)-th stage of the policy agent can be initialized by referring to that of the t-th stage as follows:

$$\begin{array}{c}{\pi }_{\omega }^{t+1}=\mu \cdot {\pi }_{\omega }^t+\left(1-\mu \right)\cdot {\pi }_{\omega }^{t-1}\hfill \\ ={\pi }_{\mathrm{ref}}^t+\mu \cdot \left({\pi }_{\omega }^t-{\pi }_{\mathrm{ref}}^t\right)\hfill \end{array}$$ 12

where μ is the forgetting rate of the t-th stage, which is set to 0.25 by default.

Referring to Eq. (7), for a query at step k + 1, we can sample a group of outputs {φ₁, φ₂, . . . , φ_N} from the old policy agent ${\pi }_{{\omega }_{old}}$ (i.e., the policy agent at the kth step), and then optimize the current policy agent by maximizing the objective function. With the reward value calculated by Equ. (11) and the updated reference model ${\pi }_{ref}^t$, the corresponding overall estimated advantage is calculated as follows:

$$E{A}^t=\frac{1}{N}\sum _{i=1}^N\frac{1}{\mid {\phi }_i^t\mid }\sum _{l=1}^{\mid {\phi }_i^t\mid }\left\{\min \left[\rho {Ǎ}_{i,l},\text{clip}\left(\rho ,1-\epsilon ,1+\epsilon \right){Ǎ}_{i,l}\right]-\kappa K{L}^t\right\}$$ 13

where N is the group size, ρ is the dynamic policy update $\frac{{\pi }_{\omega }^t\left({\phi }_{i,l}^t\mid q,{\phi }_{i,<l}^t\right)}{{\pi }_{{\omega }_{old}}^t\left({\phi }_{i,l}^t\mid q,{\phi }_{i,<l}^t\right)}$, and ${Ǎ}_{i,l}=\frac{r_i-\text{mean}\left(r\right)}{\text{std}\left(r\right)}$ is the advantage derived from the normalized reward³⁶. KL is the distribution divergence aligning to its corresponding reference model ${\pi }_{ref}^t={\pi }_{\omega }^{t-1}$,

$$K{L}^t={\mathbb{D}}_{KL}\left[\right){\pi }_{\omega }^t\mid \mid {\pi }_{ref}^t\left]\right)={\mathbb{D}}_{KL}\left[\right){\pi }_{\omega }^t\mid \mid {\pi }_{\omega }^{t-1}\left]\right)=\frac{{\pi }_{\omega }^{t-1}\left({\phi }_{i,l}^t\mid q,{\phi }_{i,<l}^t\right)}{{\pi }_{\omega }^t\left({\phi }_{i,l}^t\mid q,{\phi }_{i,<l}^t\right)}-\log \frac{{\pi }_{\omega }^{t-1}\left({\phi }_{i,l}^t\mid q,{\phi }_{i,<l}^t\right)}{{\pi }_{\omega }^t\left({\phi }_{i,l}^t\mid q,{\phi }_{i,<l}^t\right)}-1.$$ 14

With the learned policy agent and its experience pool, the target network F_θ is trained.

Accelerating Rocket on large datasets

In general, it takes a long time to get the performance value of the target model F_θ due to its time-consuming training process, especially on large datasets. As a result, it will substantially delay the return of the reward for the policy model, making policy learning inefficient or even impossible.

In fact, when we search for the optimal hyperparameter configurations, we do not concern with the absolute performance for calculating the final reward, but concern with the performance difference corresponding to different hyperparameter configurations, as shown in Eqs. (8)–(10). Consider that the distributions of subsets are positively correlated with the distribution of the whole set, especially on large datasets. We can sample a subset X_i of approximately similar distribution to that of the whole set X. We then train the task F_θ on the subset to obtain relative performance, and further calculate the approximated reward.

Therefore, we just sequentially split the whole set X into M subsets of equal size, which constitute a subset list [X₁, X₂, . . . , X_M], M=$\frac{\mid X\mid }{\mid X_i\mid }$. In each iteration of policy learning, a group of rewards are collected from s consecutive subsets (denoted as a subset span of size s) of the subset list. After that, we recurrently select the next subset span from the subset list for the next policy iteration. Figure 8c illustrates the process, and the hyperparameter optimization can be speeded up $\frac{M}{s}$ times compared to that on the full dataset.

However, we have to consider how to transfer the hyperparameter configurations on a subset to the full set. Several prior studies^57–60 have established that when the batch size is scaled by a factor of N, the learning rate should be scaled by $\sqrt{N}$. This square-root scaling rule is rooted in the principle that the gradient noise variance within a batch is equal to that of the full dataset, i.e., remaining constant under the scaling^57,58. In contrast, linear scaling — though empirically validated in refs. ⁵⁸ and ⁶¹ — lacks the same theoretical grounding, as noted even by the original authors. But in our setting, the total dataset size grows while the batch size remains fixed. Under the same constant gradient noise principle, this is equivalent to reducing the effective batch size. Hence, the learning rate should be decreased by following a square-root rule. The derivation is as follows.

Denote the total gradient noise variance of a batch B of samples as $\frac{{\sigma }^2}{B}$. For a subset, we have the total noise variance $e_i\propto {\xi }_i^2\cdot \frac{{\sigma }^2}{B}\cdot \frac{\mid X_i\mid }{B}$, where ξ_i is the learning rate on the subset X_i, and $\frac{\mid X_i\mid }{B}$ is the total training steps on X_i. Correspondingly, the total noise of the full set is denoted as $e_f\propto {\xi }_f^2\cdot \frac{{\sigma }^2}{B}\cdot \frac{\mid X\mid }{B}$. In general, we expect that the total noise variance of the subset is equal to that of the entire data set, i.e., ${\xi }_f^2\cdot \frac{{\sigma }^2}{B}\cdot \frac{\mid X\mid }{B}=={\xi }_i^2\cdot \frac{{\sigma }^2}{B}\cdot \frac{\mid X_i\mid }{B}$. As a result, we have ${\xi }_f=\sqrt{\frac{\mid X_i\mid }{\mid X\mid }}\cdot {\xi }_i$. Notably, we observe that when the ratio of the full dataset to the subset is below 10, the subset-tuned learning rate can be directly applied to the full set. Otherwise, the learning rate scaling strategy should be considered.

Implementation Details

Target tasks

We evaluate our method through comprehensive comparisons with SOTA approaches and various ablation studies.

For ablation studies, we focus on image classification to validate the key features of Rocket. For a thorough exploration of the reward dynamics, we use a subset of CIFAR-100⁶² (mini-CIFAR), comprising 5000 randomly sampled images from the original dataset. Experiments are conducted using ResNet-18 as the target network, with Top-1 accuracy as the performance metric.

For comparison with SOTAs of classical computer vision tasks, we assess Rocket’s performance across multiple tasks using large-scale benchmarks, demonstrating its broad applicability and effectiveness. Concretely,

• For image classification, we evaluate our method on the widely used CIFAR-100⁶², the Caltech-UCSD Birds-200-2011 dataset⁶³ (CUB200 for short), and ImageNet-1k (ImageNet for short) benchmarks⁶⁴. We employ three representative architectures as the target network: ResNet-18 (Res-18), ResNet-50 (Res-50)⁶⁵, and TinyVit⁶⁶ (ViT-t), with Top-1 accuracy (ACC) as the primary performance metric. The correspondence between the model architectures and the datasets is shown in Table 1.

• For object detection, we use two standard benchmarks: PASCAL VOC2012⁶⁷ (VOC_DET for short) and MS COCO2017⁶⁸ (COCO17 for short). We employ YOLOv5-n (YOLOv5 for short) and Faster-RCNN⁶⁹ (with ResNet-50 backbone, FRCNN50 for short) as the detection architecture. Performance is quantified using mean Average Precision (mAP)⁷⁰, following established evaluation protocols.

• For semantic segmentation, we evaluate our method on two established benchmarks: PASCAL VOC2012⁶⁷ (VOC_SEG for short). The representative architecture is HRNetV2p-small⁷¹ (HRNet-18 for short). Model performance is assessed using the mean Intersection-over-Union (mIoU) metric, following standard evaluation protocols.

• For optical character recognition (OCR), we select the representative method CRNN⁴⁵ for evaluation, and use word_acc_ignore_case as the metric. We use the public dataset MJSynth⁷² as the training dataset (MJTrain for short), which contains 9 million images, and evaluate the target model on six common benchmarks: CUTE80⁷³, IIIT5K⁷⁴, SVT⁷⁵, SVTP⁷⁶, IC13⁷⁷, and IC15⁷⁸.

For comparison with SOTA large language processing (NLP) models, we select several classical natural language processing tasks as follows:

• For the text-to-vector (T2V) task⁷⁹, the CoSENT⁴⁶ model is evaluated on the STS-B⁸⁰ benchmark, and the Spearman coefficient is used as the performance metric.

• For the text classification task, CLUE⁸¹ benchmarks (including AFQMC, TNEWS, IFLYTEK, CMNLI, CLUEWSC2020, and CSL dataset) are employed. The AlBert⁴⁷ backbone is used, and average accuracy is adopted to evaluate performance.

For the automatic speech recognition (ASR) task, we select popular Whisper-small⁴⁹ as the backbone (Whisper for short). The Common-voice⁸² benchmark (Co-Voice for short) is used, and the word error rate is adopted as the performance metric.

To further expand the scope of tasks, time series forecasting (TSF) is also included. TTM-r2⁴⁸ is used as the architecture. It contains 14 datasets covering weather, solar, wind, and so on. Accuracy is used for performance evaluation.

Configuration of policy agent

We adopt Qwen2.5-0.5B-Instruct⁸³ as the base architecture for both the reference model and the policy agent. The reward function’s tunable parameters β and η are initialized as 0.25 and 0.25, respectively. Both the batch size and the group size are set as 8, and the total number of stages T is set as 4. The span size is set to 8. The stage size is 16 by default.

Note that unless otherwise specified, all experiments throughout the manuscript are conducted under the same parameter configuration, including those in Table 1 and the industry case studies. For ablation studies, only the relevant parameters under investigation are adjusted, while all the other settings remain unchanged. This is to demonstrate both the general applicability of Rocket across diverse tasks and the stability of its internal configuration.

Hardware and software

All experiments are performed on an NVIDIA A800 GPU-accelerated computing system. The A800 GPU features 80 GB of GDDR6 memory, providing high-throughput parallel processing capability essential for large-scale model training. The host system is equipped with Intel Core i9 processors and 512 GB of DDR4 RAM to ensure efficient data handling and preprocessing. In practice, Rocket remains fully operational even with only 2 GPUs — one GPU for the policy agent to generate HP Configs, and another GPU is responsible for evaluating the target tasks. Note that the policy agent itself rarely constitutes the computational bottleneck; the additional GPU resources are therefore primarily allocated for accelerating the validation of target tasks.

Since MMPretrain⁴², MMDetection⁴¹, MMSegmentation⁴³ and MMOCR⁸⁴ are popular open source frameworks with friendly interfaces, we use them to train the target networks of classification, detection, segmentation, and OCR, respectively. For the NLP tasks, we use CoSENT⁴⁶ and Bert⁴⁷ as the training framework. For the TSF task, TTM-r2⁴⁸ is employed. For the ASR task, Whisper⁴⁹ is used. All random seeds are fixed.

The reinforcement learning phase is implemented using the Transformer Reinforcement Learning library (TRL)⁸⁵, a framework designed for efficient RL fine-tuning of transformer-based models. TRL is integrated with the Hugging Face ecosystem, enabling seamless compatibility with pre-trained language models and scalable reward modeling.

Author contributions

Conceptualization: Z.Z.C. Methodology: Z.Z.C., Y.Y.C,. and C.B.Z. Software: Y.Y.C. and Z.Z.C. Formal analysis: Z.Z.C. and C.B.Z. Investigation: Z.Z.C., C.B.Z., Y.Y.C, and S.G.Z. Visualization: C.B.Z. Validation: Y.Y.C. Writing and revision: Z.Z.C., S.G.Z. and C.B.Z. Supervision: X.B.L., J.H.G., F.W., S.G.Z.

Peer review

Peer review information

Nature Communications thanks Man-Je Kim and the other anonymous reviewer(s) for their contribution to the peer review of this work. A peer review file is available.

Data availability

A dataset of hyperparameter configurations containing more than 130,000 records based on ResNet-18 has been released and is accessible at https://figshare.com/s/402d2363c827a836327b. It is provided under the CC-BY-4.0 license to support research in hyperparameter space analysis. For mini-CIFAR images, we use a subset of CIFAR-100 (mini-CIFAR), comprising 5000 randomly sampled training images and 1000 randomly sampled validation images from the original dataset. It is available at https://figshare.com/s/9f23be567c0c2c0d9759. For complete Rocket training, four rounds of records on mini-CIFAR with ResNet-18 are provided at https://figshare.com/s/0f25bcf14c1e0fc0fa01.

Code availability

The whole Rocket project is licensed under the Apache License 2.0 and is available at https://figshare.com/s/eb2ba90d2ff92cd1c36d. Example training logs of Rocket for single-GPU on RTX 4070 and dual-GPU on L40 are available at https://figshare.com/s/d8f81f9b8cf38242f85b.

Competing interests

The authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at 10.1038/s41467-026-72413-9.