Machine Learning Approach to Estimating Wall-Clock Times and Assigning Resources for a Wide Variety of Quantum Chemical Calculations

Abstract

This work presents an artificial neural network (ANN) model, Jaguar Timer, which estimates the wall-clock time of quantum chemical calculationsspecifically density functional theory (DFT) and time-dependent DFT (TDDFT) calculations. The ANN model is trained and tested on data generated with the quantum chemical package Jaguar, but the same concept should be applicable to data that originate from any quantum chemical engine. We show that the model accurately predicts DFT and TDDFT wall-clock times with over 97% of test cases having an error within a factor of 4 of the observed runtime, and 77% having an error within a factor of 2. Jaguar Timer is designed to support a diverse collection of Jaguar job types and is applicable to a very wide range of wall-clock durations, from seconds to days. The model supports all atom types, basis sets, and the vast majority of exchange-correlation functionals available in Jaguar. Additionally, a wide selection of other input settings is supported, including pseudospectral (i.e., seminumerical) and analytic integrals, spin-restricted and spin-unrestricted determinants, vacuum and implicit solvent simulations, single-point energies and geometry optimizations, vibrational frequency calculations, and TDDFT single-point energies and geometry optimizations. Jaguar Timer is also capable of estimating wall-clock times of jobs executed in batched mode and jobs that employ multiple CPU cores. The wide scope of the model is its core benefit and significantly broadens the supported job and input types that have been studied in similar works. The model’s generality enables users of Jaguar to predict the length of a given set of jobs within a vast collection of common and advanced application domains. Such predictions empower users to carefully optimize their Jaguar jobs and workflows to reduce overall runtimes. In particular, we show that the Jaguar Timer estimates can be incorporated into a greedy computational resource assignment algorithm to reduce runtimes automatically within a workflow or within job batching.

1. Introduction

Density functional theory (DFT) is the most popular type of quantum chemical method applicable to small and medium-sized molecular systems. − Depending on the molecular system size and calculation settings, the computational cost of DFT can vary dramatically, with typical practical calculations taking anywhere from seconds to weeks to complete. Computational chemists are therefore faced with difficult choices when they set up calculations in order to balance the expected simulation accuracy, wall-clock time, and monetary cost against each other. For example, increasing the level of theory will typically increase the accuracy, but will also increase the wall-clock timeunless the calculation is parallelized, in which case the monetary cost will generally increase. Moreover, there are usually many specific settings exposed for any given choice of basis set and DFT functional, which can dramatically alter a job’s runtime. These settings include different algorithmic approximations, − the choice of solvation model, DFT grid size, convergence criteria, etc. The many possible combinations of these settings can lead to unexpected consequences on the length of the calculation. How can one objectively approach this challenging optimization problem without data or tools to guide the decision-making? This question is arguably even more pressing in the era of agentic artificial intelligence (AI), when these decisions often need to be made without direct human supervision.

Benchmarks in the literature can aim to provide guidelines on which DFT functionals and basis sets are appropriate for different scientific questions and the accuracy they can typically achieve. , This dimension is not the target of this work. Instead, we focus on the challenge of predicting the runtime of a particular calculation, as performed by the Jaguar quantum-chemical package. , Armed with accurate predictions of the wall-clock time for various Jaguar job types and an expectation of the accuracy each job will deliver, a scientist can make more informed decisions when launching calculations within the constraints of his or her delivery deadlines or resource limits.

As follows from the discussion above, assessing how computationally expensive a given calculation will be is a nontrivial task. To carry out this assessment, in the past, users were relying on previous experience or were spending extra time performing tests to estimate the cost of their production calculations. The first approach, which attempts to draw from experience and intuition, is abstruse even for experts and is prone to significant error. The second approach, where pilot calculations are used, is inconvenient and demands more time and computational resources. As hinted at before, the difficulty of estimating the cost stems from the very different time scales associated with different classes of density functionals, basis set choices, calculated properties, specifics of the self-consistent field algorithm, parallelization of the DFT code, nonlinear scaling of runtime with molecular system size, type and degree of integral screening, and so on. Specific implementations in diverse software packages as well as variations in hardware only increase the uncertainty. This complicated landscape of factors poses an effectively impossible challenge to DFT experts and even makes pilot job schemes untrustworthy. Furthermore, errors in predicting the runtimes of quantum chemical calculations can lead to incurring a significantly higher computational cost if the calculation winds up running much longer than initially estimated. In this scenario, the same type of job will often be launched again from scratch but with more resources or a less demanding level of theory or basis setafter the original, perhaps overly long, calculation ends up being terminated by the user. The aborted calculation may be considered a wasted computational resource. On the other hand, the DFT user may initially overestimate the cost of the calculation and choose to sacrifice accuracy in an effort to speed up the job. In this scenario, after the initial job is completed, the user may find that higher accuracy was actually affordable (and perhaps required after analyzing the results) and might be forced to submit a new calculation with different settings. Here, the initial calculation may also be viewed as a wasted computational resource. Beyond just wasting resources, however, if a DFT calculation’s cost is not accurately estimated beforehand, then there is a missed opportunity to carefully tune various job parameters to optimize the accuracy of the calculation against the cost, as the previous scenario highlights. Thus, it would be extremely valuable if practical researchers had an alternative method to generate accurate estimates of the wall-clock time of arbitrary DFT calculations of interest before these calculations are launched, without sacrificing time and effort.

Machine learning has become broadly useful across chemistry and materials science and features prominently in such topics as molecular property prediction, machine-learned force fields, and surrogate models that approximate quantities from more computationally demanding electronic structure calculations. − Here, we apply machine learning to predict the computational cost of obtaining such quantities. This strategy would lead to increased accuracy of DFT applications and increased computational efficiency by reducing wasted computational resources. Figure illustrates one such efficiency gain that is enabled by accurate wall-clock time estimates, namely, longer running jobs can be assigned more CPU cores at the expense of shorter ones, reducing the total wall-clock time of the batch.

1.

Illustration of an example optimal CPU core assignment that is achievable with Jaguar Timer’s high-accuracy wall-clock time estimates for general DFT computations. The length of the horizontal bars associated with individual jobs signifies the time necessary for the job to finish. The red color of the bar means a suboptimal assignment of CPU resources whereas the green color corresponds to an optimal situation. Through a greedy algorithm, long-running jobs are assigned more CPU cores while short-running jobs’ CPU cores are freed. These dynamic core assignments enable the full batch of jobs to finish faster than a naïve uniform assignment of CPU cores.

2. Overview of the Previous Progress

While kernel-level cost models for individual computational steps within a DFT code have been explored, Heinen and co-workers were perhaps the first to predict end-to-end wall-clock times of quantum chemical calculations using machine learning, showing that a simple Kernel Ridge Regression model can accurately estimate runtimes across a number of different job types. The work of these authors focused on wall-clock time predictions of single-point energies, geometry optimizations, and transition state searches. For each job type, separate models were generated for a few combinations of the quantum chemical method and basis set (so-called “tasks”). The predictions of this model are therefore limited to just a few types of tasks. In particular, DFT was represented by a single functional and basis combination, B3LYP/def2-TZVP. Nonetheless, using their model’s timing estimates, the authors showed that improved job scheduling can be achieved with the first fit decreasing (FFD) bin-packing algorithm. Namely, by applying FFD, the authors were able to reduce scheduler overhead and wall-clock time for a set of quantum chemistry calculations assuming that a particular number of calculations should be executed in parallel on as few nodes as possible and completed within a given maximum time duration. Interestingly, the improved scheduling was possible even though the raw wall-clock estimates were not exact. Despite the success of this work, the narrow scope of the supported tasks precludes its use in many modern quantum chemistry applications, which span a much larger space of functional and basis combinations, among many other calculation settings. Expanding the particular approach to support, e.g., other functionals and basis sets would be a tedious, expensive, and time-consuming piece of work since it would require generating a separate data set and ML model for each combination. Supporting all possible combinations in this way would be impractical.

In a later work, Ma and coworkers constructed a more transferable framework for predicting the computational time of single-point DFT and singlet excited-state time-dependent DFT (TDDFT) calculations. In their approach, Ma and coworkers similarly trained separate ML models for specific subsets of DFT functional and basis set combinations. However, their framework supported untrained combinations by scaling the prediction for an ML model that was trained on an “adjacent” functional and/or basis which was seen during the training of the model. An adjacent functional was considered to be one that existed in the same rung of Jacob’s ladder. The majority of the authors’ calculations were short in duration (CPU times of 15 min or less), with the longest running job having a CPU time of only ≈1 h. Additionally, the models only supported single-point energy calculations, making the scope of the designed tool very narrow. In practice, it would be advantageous to have cost estimates for such common job types as geometry optimizations, transition state searches, and vibrational frequency calculations while also making estimates transferable to arbitrary functionals and basis sets. The timings of quantum chemistry applications are often influenced by various other aspects of the DFT job such as the use of spin-restricted or spin-unrestricted wave functions, symmetry, different integral evaluation methods, solvent methods, the number of open multiprocessing (OpenMP) threads, CPU cores, and/or message passing interface (MPI) ranks, and so on. Furthermore, modern DFT codes can calculate a variety of properties following the converged SCF solution. An ML model or framework that does not support this broader landscape of DFT job types and settings is therefore extremely limited in its utility for modern applications.

Ma and coworkers followed their initial work with scheduling methods that employ their predicted computational runtimes. , In these works, it was shown that the described ML models could help optimize load balancing and scheduling of resources. The authors demonstrated that their ML-assisted load balancing could reduce high-throughput wall-clock times by a factor of 50–70%. In accordance with the above-mentioned development by Heinen and coworkers, this study reaffirmed that such an ML-assisted scheduling scheme could be useful for reducing computational resources, despite being built on timing predictions that are not exact.

Like in previous works, the scope of the supported DFT job types remained limited, though the authors did slightly broaden the model’s scope to include geometry optimizations and coupled cluster , jobs by using a blanket power-law scheme to transform single-point estimates into estimates for geometry optimization and coupled cluster runtimes. Specifically, CCSD(T) timing estimates were generated by squaring the timing estimates for single-point energy calculations. Similarly, geometry optimization estimates were generated by using a power of 1.25 on the single-point estimates. Both represent unreliable strategies for obtaining timing predictions. Particularly in the general case of both small and large systems, these schemes will lead to very large errors. This is because these power-law heuristics fail to account for the divergent scaling behaviors where the timings of CCSD­(T) grow as O(N ⁷), N being the size of the molecule, while DFT scales closer to O(N ^x ) where x is approximately 2–3. They also ignore the inevitable differences in the number of iterations required to optimize different molecular geometries.

3. Objectives and Scope

The present work is dedicated to the construction of a high-level computational function that makes DFT wall-clock time predictions, typically within milliseconds to seconds, for a large variety of DFT calculation types and job settings. From the outset, it is designed to support the vast majority of combinations of DFT functionals and basis sets that are shipped with the Jaguar package (on the order of tens of thousands of combinations), the only exceptions being the composite (-3c) methods , and double-hybrid functionals. , Additionally, the model incorporates many other Jaguar job types, properties, and run settings. Referred to as Jaguar Timer, it is a tool that utilizes an underlying artificial neural-network (ANN) model trained on and applied to calculations performed with Jaguar. The tool is capable of providing wall-clock time predictions in a variety of scenarios including single Jaguar jobs, batched groups of Jaguar jobs, and multidimensional coordinate scans (both distributed and in sequence). The methodology used to construct Jaguar Timer, however, is not specific to the Jaguar package, and similar tools could be developed for other quantum chemical programs.

To produce reliable predictions from Jaguar Timer’s model, we have employed advanced ML supervised-learning methods applied to a wide variety of Jaguar calculations. A diverse selection of molecules suitable for organic light-emitting diodes and organic photovoltaics, as well as small organic molecules with sizes that range from diatomics to drug-like molecules, constitutes the training chemical space. The calculation types include single-point energies (SPEs), geometry optimizations, and vibrational frequency calculations. Excited-state single-point calculations and geometry optimizations, via time-dependent DFT (TDDFT), are also supported by the model. In addition to gas-phase, solvent-phase calculations are recognized. For solvent-phase calculations, Jaguar’s Poisson–Boltzmann finite element (PBF), − polarizable continuum model (PCM), − solvation model with density (SMD), and DRACO-SMD implicit solvent models are supported. Jaguar’s TDDFT job types are also accepted as input to Jaguar Timer. The model accounts for both pseudospectral and analytic integral schemes, as well as restricted and unrestricted reference wave functions. Several tens of DFT exchange-correlation functionals representing different functional types have been employed in the model training. For more information on the Jaguar methods and properties mentioned above, see the recently published review that covers the current status of Jaguar’s feature set. From the outset, all supported job types are trained directly to actual Jaguar jobs, avoiding any reliance on heuristics.

Jaguar Timer provides an estimate for the wall-clock time of individual calculations defined only by their Jaguar job settings specified in input files. Batched Jaguar jobs (i.e., multiple independent Jaguar jobs launched in parallel) can be handled as well. Jaguar Timer reports the total time to complete all calculations in the batch along with the individual estimates for each job. Jaguar Timer can also account for multicore DFT calculationssingle jobs or batched. For batched jobs with multiple CPU cores, the tool is capable of optimizing the number of cores assigned to each job (not exceeding the total number of cores requested by the user). This estimate is available owing to the individual timing estimates that inform the greedy approach. No attempts have been made to estimate queue times when submitting jobs in high-performance computing (HPC) environments, though progress in this direction has been made by others. ,, An early, prototype implementation of Jaguar Timer has already been briefly mentioned in a review paper that focused on Jaguar. The present work reports the construction and use of a significantly more advanced version of that model that has a much wider scope of applicability. Throughout this work, the terms “walltime” and “wall-clock time” are used interchangeably.

4. Jaguar Timer Model

A technical description of the Jaguar Timer model will be given in the present section, which is split into several topics that cover the architecture, featurization, training data, and application targets.

4.1. Architecture

To our knowledge, no existing model predicts DFT computational cost across a full combinatorial space of functionals, basis sets, calculation settings, solvation models, and CPU core counts within a single architecture. This breadth of scope, however, was a key requirement of Jaguar Timer and therefore motivated its model’s design choices described below. Following prior work that used neural networks to estimate the cost of quantum chemical calculations, we adopted an ANN for Jaguar Timer’s underlying model implemented with the scikit-learn API. , The model’s architecture is a fully connected feed-forward multilayer perceptron regressor.

A tanh activation function with L2 regularization strength of α = 0.01 was applied to the output of each neuron, which was selected for its smooth, differentiable curve. Other model architectures, such as tree-based models (e.g., gradient-boosted trees), can suffer from piecewise-constant predictions that cannot accurately capture the smooth, monotonic behavior of the model’s continuous features, such as CPU core count and basis set size. See Section V of the Supporting Information for further comparisons between model architecture choices within Jaguar Timer.

The ANN architecture and activation function choice were optimized via a grid search approach which varied the hidden layer width and activation function choice. Hidden layer counts of 1, 2, and 3, and neuron counts per layer of 20, 40, 60, 80, 100, 120, and 140 were used in the grid search. The explored activation functions included the rectified linear unit (ReLU) function and the tanh function. For each point in the grid search, 6-fold cross-validation splitting was performed to improve transferability and guard against overfitting to our training set. Early stopping was utilized in cross-validation with a 15% validation holdout set to prevent overfitting. Fitting was terminated after no improvement was observed for 25 consecutive iterations. The scoring metric used for hyperparameter selection during the grid search was the negative root mean squared error (RMSE) evaluated on the log base-2 wall-clock times. This scoring metric penalizes multiplicative prediction errors equally across the full dynamic rangea factor-of-two error on a 10 s job and a factor-of-two error on a 10 h job contribute identically to the loss. This helps to ensure that the model does not sacrifice accuracy on short-running jobs in favor of reducing absolute errors on long-running ones, which would occur if RMSE were instead applied to raw wall-clock times. The model’s final architecture was selected to use 2 hidden layers with 100 neurons each. The grid search produced a broad flat basin of RMSE values across the grid points spanning hidden layer counts of 2 to 3 and 100, 120, and 140 neurons per layer. The chosen network is the smallest configuration that consistently sits within the indistinguishably best basin. After we identified the network architecture, the production Jaguar Timer model was retrained on the full training set (all training points with no held-out set) with the selected hyperparameters. The neural architecture of Jaguar Timer’s underlying model is represented in Figure .

2.

A representation of Jaguar Timer’s underlying feed-forward fully connected ANN model architecture. The input layer consists of 28 (preprocessed) features, comprising both real-valued and categorical types. The 2 hidden layers consist of 100 neurons each. The output layer is a single neuron that predicts the logarithm base-2 of the wall-clock time.

4.2. Model Input and Output Layers

A central design goal of the feature set for Jaguar Timer’s ML model was to enable accurate predictions for functionals and basis sets not seen during training. Rather than encoding each combination as a categorical label, which would fail for any label absent from the training data, we constructed continuous, physically motivated representations that transfer to unseen combinations. This subsection describes the specifics of the input layer and output layer construction and how their design choices enable widely scoped transferability.

The model’s input layer combines real-valued and categorical features. In order to encode the input structures into Jaguar Timer’s feature set, we used the molecular fingerprint associated with the electrotopological state (E-state) index from the RDKit package. , The E-state index fingerprint is a real-valued vector, where each element describes the electronic and topological environment of one of the heavy atoms in the structure. The encoding takes into account each atom’s element type, its bonding (e.g., single/double/triple/aromatic), and its attached-hydrogen count. It provides a compact descriptor of molecular composition and local connectivity that is computed quickly (microseconds). Beyond the E-state fingerprint, we also include the number of rotatable bonds and the total number of bonds to encode additional structural information in the model.

To encode the DFT exchange-correlation functional, we use a categorical bitstring representation of the functional’s characteristics, augmented with three additional real-valued features that describe the normalized computational cost of the functional. The bitstring uses 7 bits corresponding to the following Boolean properties of the functional: dispersion-corrected, long-range-corrected, hybrid, meta-GGA, GGA, LDA, and the presence of nonlocal correlation. The three additional features are obtained for every functional supported in Jaguar in the following way. For each supported functional, we perform single-point energy, geometry optimization, and frequency calculations on a test system, specifically benzene with the 6-31G** basis set.

The corresponding observed wall-clock times are then stored directly as the three additional features. These numbers provide a ranking of the systematic difference in computational cost exhibited by one functional over the other. Moreover, these features allow the model to learn functional-specific costs that cannot be captured by the bitstring alone, for example, if different implementation details are used for functionals with matching bitstrings (e.g., DFT grid size). When future DFT functionals are added to Jaguar, there will be maintenance required to compute their corresponding wall-clock times in order to support them in future versions of Jaguar Timer. However, this requires a negligible computational cost and, critically, does not necessitate the model to be retrained. This choice of functional encoding is an important ingredient that allows our model to generalize and make predictions about functionals it has not seen during training. As mentioned previously, our work goes beyond previous studies by not being limited to a particular choice of functional while avoiding explicit training on all supported functionals. While the bitstring alone offers a path to this generalizability, we found in practice that the addition of the three wall-clock features helps to achieve higher-accuracy predictions on out-of-distribution inputs.

Other categorical features that are input into the model encode the job type information. These include whether the calculation corresponds to a geometry optimization, calculates vibrational frequencies, performs a TDDFT calculation, uses the pseudospectral method for 2-electron integral evaluation, ,,,, employs a spin-unrestricted wave function, or applies an implicit solvent model. Additional real-valued features include the number of basis functions to be used for the job, the number of basis functions per atom, the number of atoms, the molecular charge, and the number of CPU cores. Although the number of basis functions per atom correlates linearly with the total number of basis functions and atom count, we retained all three features in the final model. In feed-forward neural networks, explicitly providing mathematically combined ratios, rather than requiring the network to approximate division operations implicitly, helps smooth the loss landscape and improve predictive accuracy. Physically, this ratio acts as a proxy for basis set quality and size, allowing the model to readily distinguish between large systems evaluated with smaller basis sets and small systems evaluated with larger basis sets. The full set of features is shown in Table , along with their value type and description.

1. A Summary of the Features Used by Jaguar Timer’s ANN Model .

Feature count	Feature type	Description
3	Real-valued	E-state indices (3 principal components)
1	Real-valued	Number of basis functions
1	Real-valued	Number of CPU cores
1	Real-valued	Number of atoms
1	Real-valued	Number of basis functions per atom
1	Real-valued	Number of bonds
1	Real-valued	Number of rotatable bonds
1	Real-valued	Molecular charge
1	Real-valued	Functional’s walltime of benzene single-point energy
1	Real-valued	Functional’s walltime of benzene geometry optimization
1	Real-valued	Functional’s walltime of benzene frequency calculation
7	Categorical (LDA, GGA, ...)	Bitstring that encodes functional type
2	Categorical (Gas, PBF, PCM, ...)	Solvent method
1	Categorical (Binary)	Geometry optimization
1	Categorical (Binary)	Frequency calculation
1	Categorical (Binary)	TDDFT calculation
1	Categorical (Binary)	Pseudospectral integral method
1	Categorical (Binary)	Spin-unrestricted wave function
1	Categorical (Binary)	Diffuse basis set

^aEach feature type is described, and its associated number of input features is shown. The feature counts do not account for any transformations or reductions performed using PCA.

All input feature values were transformed and scaled differently depending on their type. For general real-valued features, excluding the E-state fingerprints, we first scale the values according to the Yeo–Johnson power transform. For a given feature X ⁽ⁱ⁾for example, the number of basis functions or the number of atomsthe transform produces a new value $X_{\lambda }^{(i)}$ according to

$$X_{\lambda }^{(i)}\to \{\begin{array}{ll}\frac{{(X^{(i)}+1)}^{\lambda }-1}{\lambda }& ,\lambda \ne 0,X^{(i)}\ge 0\\ \ln (X^{(i)}+1)& ,\lambda =0,X^{(i)}\ge 0\\ -\frac{{(-X^{(i)}+1)}^{2-\lambda }-1}{2-\lambda }& ,\lambda \ne 2,X^{(i)}<0\\ -\ln (-X^{(i)}+1)& ,\lambda =2,X^{(i)}<0,\end{array}$$ 1

where λ is optimized to achieve (as closely as possible) a normal distribution. In essence, the transform applies a tunable power, λ, to positive and negative values of X ⁽ⁱ⁾ separately, stitched together smoothly at zero. By adjusting λ, features that are skewed or heavy-tailed are compressed or stretched until the distribution is approximately symmetric and bell-shaped. The two logarithmic cases are simply the limiting forms of these power functions.

The Yeo–Johnson power transformation was chosen over alternatives like Box–Cox since it naturally handles the zero and negative values present in our data set, such as molecular charge. Moreover, because DFT computation times generally scale polynomially with system size (e.g., the number of basis functions), this power transformation effectively linearizes the relationship between the right-skewed input features and the log base-2 walltime target. Additionally, for each feature index, i (not including the E-state features), we limit its values, X ⁽ⁱ⁾, to span [−1,1]:

$$X^{(i)}\to 2\frac{X^{(i)}-X_{\min }^{(i)}}{X_{\max }^{(i)}-X_{\min }^{(i)}}-1$$ 2

An application of this specific scaling after the power transformation leads to feature distributions that maintain a meaningful variance across the bounded range, i.e., preventing overly tight distributions where feature values would be difficult to distinguish. The zero-centering of the feature values also prevents the inputs from saturating the tails of the tanh activation function, thereby ensuring improved stability of the feature’s gradient. For binary features, a consistent transformation {0,1} → {−1,1} was performed.

Instead of independently scaling each E-state index by taking into account its individual extrema, we applied a global transformation across the entire E-state fingerprint set. Namely, for any specific index value X ⁽ⁱ⁾ within the complete set of E-state features {X}, the values were mapped to the [−1, 1] range by using the global minimum and maximum across the full E-state fingerprint array:

$$X^{(i)}\to 2\frac{X^{(i)}-\{X{\}}_{\min }}{\{X{\}}_{\max }-\{X{\}}_{\min }}-1$$ 3

This operation ensures that the set of transformed E-state indices preserves the relative ratios from the original molecular fingerprint. Subsequently, principal-component analysis (PCA) was performed within our training set for the E-state fingerprint to reduce the dimensionality of the E-state features, of which there were 79. Only the 3 most dominant components were used as inputs to the model which captured 75% of the total variance of the training set. While increasing the number of components to 10 showed a modest improvement in training RMSE (from 0.60 to 0.55), we chose 3 to limit the risk of overfitting. Keeping a higher-dimensional representation of the E-state fingerprint exposed the model to severe out-of-distribution instabilities which resulted in erratic predictions when we evaluated test molecules containing local atomic environments underrepresented in the training data. By compressing the fingerprints into 3 principal components, we inherently regularize the network against this vulnerability. Furthermore, this fairly aggressive dimensionality reduction mitigates the “curse of dimensionality”, allowing the model to achieve robust generalization with a smaller set of training structures than would be required to adequately sample the full E-state feature space. Permutation importance analysis confirmed that E-state components consistently rank among the least influential features in the model, heavily overshadowed by macroscopic parameters such as the number of basis functions, job type (e.g., single points, optimizations, frequencies), and CPU core count.

One-hot encoding is used on the categorical feature set to avoid artificial ordering of the feature values. For example, the solvent method choice is one such categorical feature that benefits from one-hot encoding. The solvent method choice is broken into 2 binary features for the PBF and PCM models. A gas-phase calculation is represented by both of these binary features being set to zero. Note that Jaguar Timer maps the SMD and DRACO-SMD solvation models to the PCM result due to significant similarities that these methods’ algorithms and computational costs share with those of PCM. Consequently, all PCM, SMD, and DRACO-SMD inputs yield identical wall-clock time estimates from Jaguar Timer. The full set of features was chosen to encode the given input geometry, system size, number of CPU cores, and the various calculation types that are supported by the model. In selecting Jaguar Timer’s features, we used the feature permutation protocol to determine the importance of each potential feature that we considered inputting into the model.

The output layer consists of a single neuron, targeting the base-2 logarithm of Jaguar’s wall-clock time. Using a base-2 logarithmic target compresses the large scale of the training data (spanning seconds to days) which leads to improved model performance overall. Additionally, this logarithmic transformation prevents the model from disproportionately penalizing errors on longer running jobs, which would artificially dominate the loss function if raw wall-clock times were used instead. Consequently, during training and hyper-parameter optimization, Jaguar Timer’s model accuracy was tested against observed wall-clock time on the same base-2 logarithmic scale. For practical applications, we desired the estimates to deviate by less than a factor of 4 (on a linear scale) from observed wall-clock times. While a deviation of ±2 on a base-2 logarithmic scale is seemingly large in isolation, it translates to a strict tolerance when compared to the model’s large dynamic range of roughly 18 powers of two (from seconds to days). For example, for a job that finishes in 5 min, we desire Jaguar Timer’s estimate to fall inside the range of 1.25 to 20 min. Similarly, for a 5 h job, the desired estimate range is between 1.25 and 20 h. As can be seen from these examples, the model is not designed to pinpoint the number of minutes or hours the calculation is going to take. It rather focuses on predicting “the order of magnitude” accuratelywhether the calculation will finish in a matter of seconds, minutes, hours, days, or weeks. This level of accuracy is expected to be more than sufficient for providing a useful way for users to plan their application targets and tune their job settings for optimal performance and accuracy.

4.3. Training Data Set

The training data set comprises 12,500 Jaguar calculations generated by uniformly sampling the model’s supported job types and input parameters. To construct this set, we curated a diverse baseline of 66 chemical structures that uniformly span a size range of 1 to 100 atoms, encompassing organic, inorganic, and drug-like molecules, as well as isolated free atoms. For each structure, calculation settings were randomly drawn from the available categorical features. To ensure representative coverage while maintaining computational tractability, the sampled density functionals and basis sets were restricted to a highly diverse subset of the fully supported set provided by Jaguar. The set of functionals and basis sets used in training is shown in Table . The number of CPU cores was also selected at random from 1, 2, 4, 8, 12, and 16.

2. A Summary of the Functionals and Basis Sets Used in Training and Testing Data .

Functionals
Training		Testing
B3LYP	MN12-SX-D4	B3LYP-D3
B3LYP-D3	MN15	BLYP
BHandH	PBE	CAM-B3LYP
BLYP	PBE-D3	M06-2X
BLYP-D3(BJ)	PBE0	M06-L
CAM-B3LYP	rSCAN-D4	M11
HF	SCAN0	PBE
HSE06	SVWN5	PBE0
HSE06-D4	TPSS	TPSS
M06-2X	ωB97X	ωB97X-D
M06-D3	ωB97X-D	GLYP-D4
M06-L	ωB97X-V	B3PW91
M11		SOGGA11
M11-L		uPBE0

Basis sets
Training	Testing
STO-3G	STO-3G
6-31G**	6-31G**
D95	cc-pVDZ++
def2-TZVP(-f)	6-311G*
DZVP	LACVP**+
LACVP**	3-21G*
cc-pVDZ++	6-31+G*
cc-pVTZ	D95**
cc-pVQZ	cc-pVDZ
	cc-pVTZ(-f)

^aThe name of the basis set X­(-f) denotes basis with f and higher angular momenta functions removed. Blind tests for functionals and basis sets were included to verify the transferability of the Jaguar Timer model.

^bNot used during training of the Jaguar Timer model.

To assess the sufficiency of the set of 66 structures used in training, we undertook a learning curve analysis in which the model was retrained on subsets of increasing numbers of structures and diversity. We varied the number of structures from 5 to 65 and evaluated the accuracy of each model on our blind test set (see Section III of the Supporting Information). For each structure count, we trained 5 individual models using randomly selected structures from the training set. The test’s RMSE decreased rapidly up to 20 structures but then became shallow beyond 35. The marginal gains from increasing the structure count beyond 66 did not justify the added computational cost to include more structures in the training set.

4.4. Jaguar Timer’s Supported Job and Workflow Types

The Jaguar Timer tool is designed to accept single or multiple Jaguar input files which define the atomic coordinates of the molecule, calculation settings, and basic job parameters (e.g., the total number of CPU cores). It then outputs an estimate for the wall-clock time after transforming the base-2 logarithm estimate from the underlying neural network. When multiple Jaguar input files are provided, it is assumed that the user will launch the jobs via Jaguar’s batching workflow, where the total number of CPU cores will be distributed across the set of inputs and then launched simultaneously. For these batched cases, Jaguar Timer’s estimate is based on one of two modes. The first mode assumes that the CPU cores are distributed to the jobs in a round-robin fashion. An alternative mode allows Jaguar Timer to use its individual estimates for each job to distribute the CPU cores optimally such that the total wall-clock time of the batch is minimized. The greedy algorithm used for this dynamic distribution of CPU cores is detailed in Section . The tool is also capable of scheduling computational resources for the coordinate scan workflow which is a standard calculation type available from Schrödinger software distributions. After determining the number of scan points and corresponding geometries, the model is able to build a timing estimate of the full scan from the individual scan point estimates.

As detailed in Table , Jaguar TImer supports an exceptionally comprehensive range of calculation parameters. To our knowledge, the scope of Jaguar Timer significantly surpasses that of the existing machine learning models designed to predict the computational cost of quantum chemistry codes. By accounting for the combinatorial explosion of input settings, Jaguar Timer supports in excess of one million unique calculation configurations for any given structure, assuming fixed charge and spin multiplicities.

3. All Execution Types and Input Settings Currently Supported by the Jaguar Timer Model .

Category	Supported Job Types and Settings
Fundamental job types	• Single-point energies (SPEs)
	• Geometry optimizations
	• Vibrational frequency calculations (with or without an initial geometry optimization)
	• Time-dependent DFT (TDDFT) calculations
Solvation models	• Gas-phase (vacuum) simulations
	• PBF solvent model
	• PCM solvent model (with or without DRACO radii)
	• SMD solvent model (with or without DRACO radii)
Integral methods	• Pseudospectral (seminumerical) integrals
	• Analytic integrals
Wave functions	• Spin-restricted reference wave functions
	• Spin-unrestricted reference wave functions
Functionals and basis sets	• All atom types, basis sets, and DFT exchange-correlation functionals (excluding -3c methods and double hybrids) available in the Jaguar package
	• Supported DFT functional types include LDA, GGA, meta-GGA, hybrid, dispersion-corrected, long-range-corrected, and nonlocal correlation
Execution types	• Single Jaguar jobs
	• Batched Jaguar jobs (multiple independent jobs)
	• Multicore Jaguar jobs (single and batched)
	• Multidimensional coordinate scans (both distributed and in sequence)

^aIn combination, for a given structure (fixed charge and spin multiplicity) and excluding execution types, Jaguar Timer supports over million Jaguar job types.

5. Results

We have tested the accuracy of the Jaguar Timer model on a widely varied selection of molecular structures, job types, and input settings. In order to validate the model’s transferability, we generated a “blind” test set of Jaguar calculations that span the range of Jaguar Timer’s supported Jaguar job types. All structures used for this independent test were not utilized during the training of the model. For each structure, we uniformly sampled input settings from all supported job types. This test set featured a varied subset of basis sets and functionals. The functionals used for the testing set were chosen similarly to those from the training setspanning over GGA, meta-GGA, hybrid, and long-range corrected hybrid functional types. Basis sets were also chosen to span a large range of sizes. In order to probe the transferability of the model to unknown levels of theory, a subset of functionals and basis sets which were not included in training was added to the testing set. The sampled basis sets and functionals present in this test are shown in Table . The number of CPU cores used for each job was also uniformly sampled from 1, 2, 4, 8, 12, or 16 CPU cores. By construction, this testing set uniformly covers a wide variety of Jaguar input types and combinations of settings. This approach prevents the testing set from being more heavily distributed in certain regions of feature space over others. The final testing set consisted of 126 diverse molecules, which were distributed among 786 Jaguar jobs.

Figure demonstrates Jaguar Timer’s accuracy on the “blind” test set, where all job types are sampled. The x-axis in the plot brings out the wide range of time scales that are available to the modelfrom a few seconds to multiple days. The accuracy over the full-time range spanned by the set of tested job types remains essentially constant.

3.

Correlation between Jaguar Timer’s estimates and observed wall-clock times for all calculation types (786 jobs) on the left and the corresponding histogram on the right. The Jaguar jobs cover a diverse set of quantum chemistry calculation types and molecular system sizes. The job types include single-point energy calculations, geometry optimizations, excited-state single points and geometry optimizations with TDDFT, and vibrational frequency calculations. Both gas-phase and implicit solvent calculations are included for all job types. The pseudospectral and fully analytic integral methods are also uniformly represented in the set of calculations. The calculations have been carried out on 1 to 16 CPU cores.

One can see that the majority of predictions (77%) fall within one power-of-2 error relative to the observed runtime. Additionally, nearly all cases (97%) have an error that is less than 2 powers-of-2 (i.e., less than 4).

The root mean squared error (RMSE) on the blind test set is 0.87 on a log base-2 scale (an average multiplicative error of 2^0.87 ≈ 1.83). In comparison, the RMSE on the training set was 0.60 (a multiplicative error of 2^0.60 ≈ 1.52). The moderate increase in the error when moving to the blind test set reflects its diversity, which includes molecular structures, functionals, and basis sets not seen during training.

This is a high level of accuracy for such a widely scoped ANN model that covers a broad range of wall-clock times and a diverse collection of Jaguar job types and run settings. Judging from these results, Jaguar Timer is the first ML model that is capable of generating accurate timing estimations for such a varying set of QC calculations. The right-hand side of Figure is a histogram of the ratio of predicted wall-clock times to the observed wall-clock times on a base-2 logarithmic scale. The histogram shows that the errors are approximately normally distributed around the observed wall-clock times with a mean (μ) ratio of 1.09 and a standard deviation (σ) of 1.81, implying that ≈68% of the test estimates fall within a factor of 1.81 from the observed results. For the multijob cases, namely, batched jobs and coordinate scans, the overall wall-clock time estimates are assembled directly from the single-job predictions. This therefore leads to errors for multijob timings that are no more than the individual errors of single jobs.

Figure shows that Jaguar Timer has internalized the dominant physics-based scaling laws of DFT and captures the expected trends in the computational cost that were not explicitly targeted in the model training. Below, we discuss this in more detail with examples.

4.

Analysis of molecular structures (top) and Jaguar Timer’s corresponding scaling and accuracy as the number of CPU cores and the basis set size are varied for single-point energy calculations [(a) and (b)] and geometry optimizations [(c) and (d)].

Two scaling variables are probed on the x-axis: (1) CPU core count (Figure a and c), and (2) basis set size (Figure b and d). Two job types are used in this test, single-point energies and geometry optimizations, and four representative molecules (top row of Figure , A–D, spanning small to large system sizes). For the CPU core dependence, the model reproduces both the qualitative decrease of Jaguar’s speedup with increasing core count. This sublinear scaling is an expected consequence of Amdahl’s law and is the universal behavior of parallelized quantum-chemistry codes. , Also, the model accurately reproduces the molecule-specific magnitude of the CPU core speedup, where the largest test molecule (C) reaches a higher 16-core speedup compared to the smaller examples (A, B), the speedups of which saturate earlier. Panels (b) and (d) probe basis set scaling. Across the STO-3G, 6-31G*, 6-31G**, 6-31++G**, a def2-TZVP series, the model reproduces the steeply growing observed walltimes, and therefore the dominant polynomial growth, as the number of basis functions increases. If one compares the single-point subfigures (a,b) to the geometry optimization subfigures (c,d), the predicted vs observed agreement is visibly better for single-point energies. This is consistent with the RMSE analysis in Section I of the Supporting Information, where geometry optimization timings carry larger residual error attributable to the additional uncertainty in optimization step counts.

All training and testing data were generated on a dual-socket server with two AMD EPYC 7313 processors (16 cores per socket) clocked at a 3.0 GHz base frequency. This is a reasonable and representative platform for the purpose of this study. The EPYC 7313 is a mainstream processor typical of what users often encounter in HPC clusters. Since the EPYC 7313 is neither unusually fast nor slow, the model is calibrated to a realistic middle ground for Jaguar CPU calculations. The model’s absolute estimates could obviously show large errors for substantially different hardware. However, the relative wall-clock times between jobs (and consequently the optimized CPU core assignments derived from them, which we discuss in Section ) are largely hardware-independent.

While the absolute estimates serve as useful guides on any platform, the core assignment algorithm discussed below and its associated speedups are expected to transfer across hardware types without the need to retrain the model.

One could make the ML model cognizant of the hardware it is executed on and for which it is requested to generate predictions. For example, it could time the execution of a tiny standard piece of code and then “calibrate” itself. While this is certainly possible in principle, some practical considerations deprioritized this line of development for the time being. One such practical complication is that the user often sets up calculations on one kind of hardware (such as a personal desktop computer) but executes them on another (such as a compute cluster). Establishing communication between the two pieces of hardware in the context of Jaguar Timer requires additional engineering work preceded by the design of an appropriate user interface. This separate effort, not essential for the development of the scientific ideas exposed in the present work, might be attempted by us in the future.

Given that Jaguar Timer can accurately predict the timings of many different DFT job types, we conjecture that it will be generally useful for quantum chemists in planning and scheduling their applications, as most practical DFT applications are supported. In the next section, we show how Jaguar Timer’s estimated wall-clock times can be incorporated into a CPU core assignment protocol to reduce the total runtime of DFT batches or workflows.

6. Discussion

In the following subsections, we outline various ways in which Jaguar Timer’s wall-clock time estimates are used in practice. The examples covered include enabling users to balance the accuracy and runtimes of their DFT computations and therefore limit wasted resources and consequently speed up their scientific studies. Additionally, timing estimates can be used to more precisely inform job schedulers about the expected runtime of a particular job or batch of jobs, which can improve the overall efficiency of a queue via more optimal backfilling. We also discuss how the total walltime needed to complete a batch of jobs can be reduced by using Jaguar Timer to optimize the distribution of a fixed pool of resources.

6.1. Informed Choice of Settings and Scheduling

With the aid of the Jaguar Timer wall-clock time estimates, users performing Jaguar calculations have the potential to appreciably speed up the time required to complete their applications. Alternatively, the tool’s estimates can potentially help the user in selecting job parameters promising higher accuracy before launching the job, provided that the user is willing to accept the additional cost of the final high-accuracy calculation. The ability for users to optimize their Jaguar job settings in this way, to balance accuracy and speed, is a key strength of the tool. Informed choices can be made about the application’s level of theory, desired properties to compute, parallelization, and so on, when the expected cost is provided for each choice essentially instantaneously. This cost-balance strategy in Jaguar is made possible by Jaguar Timer’s broad coverage of Jaguar job types. Similar tools, which have so far been much narrower in scope (see references in Section ), do not enable the same level of flexibility in selecting application parameters.

Additionally, once application parameters have been selected, Jaguar Timer estimates can potentially benefit the user further. Namely, in high-performance computing environments which rely on queues and job schedulers to manage resources, having accurate wall-clock time estimates gives users the ability to define reasonable time limits for their queue submissions with ease. Imposing these more accurate time limits when submitting jobs can significantly reduce the amount of time the user’s jobs wait in the queue by giving the scheduler more accurate timing information about the jobs. This is especially helpful for backfilling, where shorter jobs can start earlier than otherwise to fill queue slots and improve resource utilization. Without accurate knowledge of a job’s wall-clock time, users might overestimate the queue time limit for their job, which can then cause long delays in the queue.

6.2. Optimized CPU Core Assignments

With accurate wall-clock time predictions from Jaguar Timer, batched DFT calculations can be ordered and assigned CPU core counts rationally to minimize the total wall-clock time. Specifically, the problem we intend to optimize is when given a batch of Jaguar jobs and a number of cores to consume simultaneously, what distribution of the cores is optimal such that the total wall-clock time of the batch is minimized. It might not be obvious to users how to assign CPU cores to subjobs, a priori, in order to minimize wall-clock time, especially considering the wide variety of possible calculation types. For this reason, we devised an algorithm based on Jaguar Timer estimates to optimize the CPU core assignments automatically given the number of cores and the set of Jaguar jobs to be batched. The optimized CPU core assignments allow users to achieve a desired wall-clock time with fewer resources compared to a less intelligent assignment algorithm, e.g., uniform CPU core assignments (so-called round-robin). Other resource optimization tasks could be explored as well, similar to Heinen’s bin-packing solution for batch submissions that require a fixed set of nodes (or cores). Here, we have focused on the use case of queuing all the batch’s jobs individually and optimizing the cores allotted to each job assuming a fixed set of cores since many workflows utilizing Jaguar submit jobs in this fashion.

To allocate CPU cores automatically, we use a greedy algorithm to assign core counts to the individual Jaguar jobs constituting a batch. Namely, we begin by assigning a single core to all jobs since all jobs will need at least one core to run on. We then assign an additional core to the longest running job following Jaguar Timer’s estimates with current assignments. We continue to assign cores in this way until all requested cores have been assigned. For each round of assignments, the model’s estimates are updated on the basis of the present CPU core counts. As shown previously, Jaguar calculations scale well up to 8 cores but perform worse at 16 cores or more. For this reason, we limit the number of cores for any given job to 8. This procedure ensures that the longest running jobs will have the most CPU cores assigned to them (up to 8) and the shortest running jobs are not using more cores than necessary. CPU cores will be assigned uniformly if all jobs are predicted to run for approximately the same amount of time. The procedure also ensures that the requested core count is honored for a longer duration of the batch’s runtime. That is, we avoid the scenario where, say, a single long-running job dominates the total runtime of the batch and is running with far fewer cores than were originally requested. Limiting the number of cores assigned to any given job also helps avoid CPU hour costs from growing disproportionately relative to the walltime speedup of the batch. See Figure for pseudocode describing the CPU core assignment algorithm.

5.

Greedy algorithm for optimal CPU core assignment using Jaguar Timer estimates.

Figure demonstrates the speedups achieved by our optimized core assignments. A comparison is made with the naïve uniform allocation of resources for a collection of job batches. The collection includes 50 batches that span from 4 to 36 Jaguar jobs. The individual Jaguar calculations all employ the B3LYP-D3/6-31G** level of theory and contain anywhere from 195 to 829 basis functions. Job types are uniformly sampled from single-point energies, geometry optimizations, frequency calculations, and geometry optimizations with frequency calculations. Excited-state single-point and geometry optimizations, through full linear-response time-dependent DFT, are also sampled. The use of the pseudospectral method is also uniformly sampled for all Jaguar jobs. For each batch, the uniform assignments utilize 2 CPU cores per job, and batches using optimized assignments utilize an identical core count.

6.

The Wall-clock times (a) and speedups (b) of a wide collection of batched jobs using Jaguar Timer’s optimized CPU core assignment algorithm relative to assigning CPU cores uniformly. For the uniform case, 2 CPU cores are assigned to all individual jobs in each batch. The optimized case uses the same total number of cores in the batch. In (b), the added CPU hours cost is also shown for the optimized assignments relative to uniform, as a ratio. In (c), a comparison of the speedups relative to the CPU hours cost is shown.

Shown in Figure a are the walltimes for each individual batch for both the uniformly distributed CPU cores and the Jaguar Timer-optimized distribution. The batches are sorted by batch walltime from shortest to longest. The figure gives a broad view of the performance of the optimized assignments for a rich variety of batches. For 49 out of 50 cases, the optimized assignments lead to a reduction in the total runtime of the batch. In Figure b, the speedup of the optimized cases is shown relative to the uniform case. To quantify the inhomogeneity of each batch, we calculated the coefficient of variation, δ _V , of the constituent job walltimes which is the ratio of the standard deviation σ and the mean μ,

$${\delta }_V=\frac{\sigma }{\mu }$$ 4

Here, δ _V serves as a measure of batch heterogeneity. When δ _V is close to zero, it represents a homogeneous batch of jobs (all with similar wall-clock times) and its high value indicates a more diverse workload containing a mix of short- and long-running Jaguar jobs. The δ _V values are calculated on the basis of the observed wall-clock times from the uniformly distributed case. The figure illustrates the performance gains of the optimized strategy relative to the uniform baseline. Jaguar Timer appropriately allocates more CPU cores to more demanding jobs, preventing massive individual jobs from bottlenecking the entire batch. Conversely, as batch heterogeneity decreases, the speedup ratio tends toward 1.0, correctly reflecting the fact that uniform resource distribution is trivially near-optimal when job sizes are more homogeneous. We also see that the single case where the optimized strategy did not decrease the walltime is the most homogeneous batch. This indicates that the batch was homogeneous enough such that uniform CPU assignments were likely already close to optimal.

Jaguar does not exhibit perfect parallel efficiency. Hence, when one increases the number of CPUs to reduce the walltime of a single job, the total number of consumed CPU hours increases slightly. We therefore also show (via the orange circles) in Figure b the cost incurred when optimizing the CPU distribution as the ratio of the total CPU hours used in the optimized versus uniform scenarios. Crucially, one can see that the dramatic reduction in wall-clock time does not incur a significant penalty in the cost of computational resources. Increasing the core counts does introduce parallelization inefficiencies, but the cost penalty remains largely stable with only a very weak correlation with batch complexity (Pearson coefficient r = 0.226). As previously suggested, the limit of 8 cores set on individual Jaguar jobs helps to keep computational costs low given that Jaguar scales well up to this point for all system sizes and job types. It is worth noting though that the trend of the CPU hours cost is a consequence of the specifics of how Jaguar DFT calculations scale with the number of CPU cores. Finally, Figure c also shows how the speedup compares to the CPU hours cost. This speedup is also plotted against the δ _V of the batches. We similarly observe a positive trend, where the more nonuniform batches are able to achieve higher efficiency relative to more homogeneous batches.

Table gives an explicit example of Jaguar Timer’s optimized CPU assignments for a single batch of subjobs with highly varied walltimes. The example was selected from the same set of batched jobs that comprised Figure . For this example, we selected a more nonuniform batch which led to a sizable speedup over round-robin assignments for a clearer demonstration of how the batch benefits in practice. For the given batch of 6 jobs, the individual job settings were selected randomly from single-point energies, geometry optimizations, and/or vibrational frequency calculations. All subjobs were randomly chosen to proceed either fully analytically (in 4-center integrals) or with the pseudospectral approximation (seminumerical integrals). All calculations were carried out with the B3LYP-D3/6-31G** level of theory. It follows from Table that the wall-clock time of the job batches varies substantially, spanning an interval from 37 s to 9.7 h. The jobs also span all possible job types, from single-point energy calculations up to geometry optimizations followed by frequency calculations.

4. An Example Jaguar Batch with and without Optimal CPU Assignments .

Subjob index	Basis functions	Assignment	Integrals	Job type	Uniform walltime, s	Optimized walltime, s
01	239	1	PS	geopt. + freq.	1790	2282
02	780	6	PS	geopt. + freq.	35026	11911
03	370	2	analytic	geopt. + freq.	5500	5235
04	289	1	analytic	geopt.	1299	1896
05	550	1	PS	freq.	2389	3641
06	455	1	PS		37	87
Total walltime (h)					9.73	3.31 (2.94 × speedup)

^aIn this example, six subjobs were launched in the batch with a pool of 12 CPUs in total. For each individual Jaguar job, 8 trial runs were launched and the minimum wall-clock time from the trials was taken. This was done to reduce the impact of run-to-run noise in the wall-clock time due to possible heavy load on the compute node, etc. The jobs varied randomly in size from 239 basis functions up to 780. The job types also varied among single-point energies (shown without job type labels in the table), geometry optimizations (geopt.), frequency calculations (freq.), and geometry optimization followed by frequency calculation (geopt. + freq.). The use of the pseudospectral integral method (PS) or the fully analytic method (analytic) is also randomly chosen throughout the batch of jobs. The table compares the full runtime of the batch with uniform assignments (all jobs assigned CPUs) versus the optimized assignments generated on the basis of Jaguar Timer wall-clock time estimates. In this particular example, the optimized assignments lead to a speedup of 2.94 which is a 66% savings on the unoptimized wall-clock time.

Finally, the use of optimized CPU assignments generated on the basis of Jaguar Timer’s walltime estimates enables users to directly reduce the monetary cost of their Jaguar batch job submissions. For example, the user can target a desired total wall-clock time by using fewer cores than would be needed if round-robin assignments were applied. In the latter case, all jobs would need to (wastefully) utilize approximately the same number of CPU cores as the longest running job. However, with optimized core assignments, the user can request fewer cores in total while still ensuring the longest running jobs get higher core counts to achieve the desired runtime.

6.3. Case Study: Large Optimization Batch of Naturally Occurring Products

As a realistic application of the Jaguar Timer optimal assignment algorithm, we curated a data set of somewhat unusual 119 marine- and plant-based molecules (for representative reviews of such molecules, refer to refs , ) which we then optimized at the B3LYP-D3/6-31G** level of theory in one large batch. These molecules, by virtue of peculiarities in their structure, such as multiple bromine atoms concentrated in a small molecular fragment, or a chain of sulfur atoms connected to an aromatic ringsee Figure for examples from the test setbelong to regions of chemical space that were not exposed to Jaguar Timer during training.

7.

A representative sample of naturally occurring compounds showing the diversity and complexity of the full data set on which a test of Jaguar Timer was performed. The full test set comprised 119 molecules.

An application of this optimization job would be to subsequently use NMR spectroscopy to categorize unusual molecules and conformations generated from marine and plant organisms. Furthermore, the data set could be useful in validating and improving NMR modules for predicting chemical shifts for such compounds. The molecules span a range of system sizes from 10 atoms to 195 atoms, or 140.3 to 1271.6 Da. Over such a large range, the optimal CPU core assignment algorithm is expected to enable a substantial speedup for the overall wall-clock time of the full batch. As such, the set corresponds to a key stress test of Jaguar Timer and the CPU core assignment algorithm built from its estimates. To investigate the speedup, we launched the batch of optimizations with uniformly distributed CPU cores and, separately, optimized CPU core assignments using the algorithm described in Section . We used a total of 238 cores for both batches. Figure provides a comparison of walltimes per job between the two assignment schemes. The jobs in the figure are sorted in order of decreasing wall-clock time. For clarity of presentation, only the longest running 41 jobs are shown according to the uniform assignment walltimes, though 119 jobs were launched in each batch. Of all jobs discarded, the wall-clock times were less than 2 h, and the optimized algorithm appropriately assigned only a single core to these discarded shorter jobs.

8.

Wall-clock times per job of a single batch of geometry optimizations over a heterogeneous set of naturally occurring products. Both optimized CPU core assignments with Jaguar Timer and uniform assignments are shown. Only the longest running 41 jobs in conformity with the uniform assignment walltimes are shown, though a total of 119 jobs were launched in the batch. The gray bars correspond to the uniform CPU core allocation with 2 cores per job. The multicolored bars correspond to the Jaguar Timer optimized assignments with the same total number of cores. The jobs in the figure are ordered from longest to shortest according to the uniform wall-clock times. Overall, the optimized assignments reduce the total walltime from 57.2 to 19.2 h, a speedup of 3.0×. Meanwhile, the computational cost is only moderately increased from 744.0 CPU hours to 891.7 CPU hoursan increase of just 20%.

The gray bars correspond to the baseline uniform allocation where every job is assigned 2 CPU cores. Uniform assignments lead to a total wall-clock time of 57.2 h. Shown in multicolored bars are the Jaguar Timer-optimized assignments using the same total number of CPU cores across the batch. The Jaguar Timer algorithm correctly identifies the most computationally demanding tasks and assigns them the maximum allowable resources, drastically reducing their individual runtimes. Conversely, shorter calculations are throttled to a single core, extending their local runtimes but ensuring they do not consume resources needed by larger jobs. This redistribution removes the bottleneck and reduces the overall wall-clock time to 19.2 h, which corresponds to a 3.0× speedup. The computational cost is only moderately increased from 744.0 CPU hours to 891.7 CPU hours (an increase of just 20%). The moderate increase in the CPU hours cost is a specific consequence of how Jaguar’s performance scales with the number of CPU cores. By allocating only a maximum of 8 cores, we guarantee that all jobs remain in a core count regime where the scaling is favorable, and therefore the impact on CPU hours cost is marginal. In summary, these encouraging results over a set of diverse and unusual structures showcase the degree of transferability achievable by the Jaguar Timer model: the model is able to determine the relative cost of the jobs and assign CPU cores appropriately. This transferability of the model is confirmed further in Supporting Information, where we show a stress test of the model’s accuracy by using a collection of molecules that contain less common elements (which were not present in the model’s training set). The results in Figure do reflect the fact that Jaguar Timer’s predictions are not exact. For instance, in hindsight, we can argue that even more cores could have been shifted to the top 16 jobs or so. Note, however, that doing so would not impact the overall walltime of the full batch since the largest bottleneck jobs already have the maximum number of cores assigned. This is further evidence that Jaguar Timer is sufficiently accurate to be a useful tool in optimizing the distribution of resources.

7. Conclusion

We have shown in this paper that Jaguar Timer, a computational tool that utilizes a simple ANN model, is able to generate high-accuracy predictions of wall-clock times for DFT calculations performed with the Jaguar quantum chemistry package. Measured against a blind testing set encompassing all supported Jaguar job types, the model achieves an RMSE of 0.87 on a log₂ scale. Using a log₂ scale, the model is able to accurately estimate total walltimes across a very large dynamic rangefrom seconds to multiple dayswhile avoiding bias toward long-running jobs. The model’s predictions were shown to agree with observed wall-clock times within a factor of 4 error for 97% of test cases and within a factor of 2 error for 77% of test cases. Moreover, the accuracy of the model is consistent over a very large range of Jaguar runtimes, from a few seconds up to a few days.

A key strength of the Jaguar Timer tool is its broad coverage of Jaguar calculation types. By supporting all of Jaguar’s DFT functionals and basis combinations, along with the ability to estimate the computational cost of single-point energy calculations, geometry optimizations, TDDFT, and vibrational frequencies, we believe the tool can serve as a valuable resource for many real-world production-level applications that utilize the Jaguar quantum chemistry package. Beyond Jaguar, the framework should be transferable to other quantum chemical packages for developing tailored ML solutions that predict wall-clock times and optimize computational resources, since the design of our tool is agnostic to the underlying package used.

In Section , we demonstrate how Jaguar Timer’s estimates can be used in practice to significantly reduce both time-to-completion and monetary cost of computational workflows and applications. We discuss how this can be achieved by letting the user make more informed choices and hence avoid wasting time and money on repeated or exploratory job submissions. Additionally, by accurately predicting the scaling of Jaguar walltimes with CPU cores, the model enables optimized CPU core assignments for batched Jaguar jobs in high-performance computing environments. We show an example where the optimized assignment of a given number of total CPU cores leads to overall speedups of 3–5× (a savings of 66–80%) for a batch of inhomogeneous Jaguar jobs compared to a uniform assignment. We also show how optimized nonuniform CPU assignments can provide these speedups with only a modest increase in total monetary cost, taking into account that Jaguar (like many packages) does not exhibit perfect parallel efficiency when running on multiple CPU cores. In contrast, we also note a scenario where Jaguar Timer allows the user to explicitly use fewer cores with optimized CPU allocations to achieve a target wall-clock time for a batch of jobs, and hence use fewer CPU hours in total, thereby reducing the monetary cost. Finally, we highlight how other resource optimizations also benefit from the tool’s accurate wall-clock time predictions, e.g., informing the setting of time limits for calculations submitted to queues which enables resource schedulers to optimally backfill.

Given the success of this work, we plan to build upon the current Jaguar Timer model to make it aware of other Jaguar settings and job types not included in this work: levels of theory such as machine-learned force fields and resolution-of-identity perturbation theory (RI-MP2) as well as concrete workflows that utilize Jaguar such as Jaguar pK _a, AutoTS, E-sol, an atropisomerism prediction workflow, and others. Expanding the model in these directions would provide automated runtime prediction and resource allocation for an even broader range of Jaguar calculations and production workflows. This would further improve resource utilization, reduce batch completion times, and facilitate more efficient, large-scale computational studies.

The software technologies used for this study are available at https://www.schrodinger.com/downloads/releases and are subject to Schrödinger software licensing.

The Supporting Information is available free of charge at https://pubs.acs.org/doi/10.1021/acsomega.6c07026.

• Additional comparisons of measured and predicted timings, details on training of the ML model, as well as the atomic coordinates of the molecules used for the model’s training and testing (PDF)

• Coordinates (ZIP)

M.C.B.: Conceptualization (lead); Programming (lead); System design (lead); Data collection (lead); Analysis (lead); Writing and editing (lead). M.A.W.: Conceptualization (supporting); System design (supporting); Writing and editing (supporting). A.D.B.: Conceptualization (supporting); System design (supporting); Writing and editing (supporting).

The authors declare the following competing financial interest(s): All authors have a financial interest in Schrödinger, Inc. and its products.