Fig. 1. The workflow of deep RL algorithm for generating new SMILES strings of compounds with the desired properties.

(A) Training step of the generative Stack-RNN. (B) Generator step of the generative Stack-RNN. During training, the input token is a character in the currently processed SMILES string from the training set. The model outputs the probability vector p_Θ(a_t|s_{t − 1}) of the next character given a prefix. Vector of parameters Θ is optimized by cross-entropy loss function minimization. In the generator regime, the input token is a previously generated character. Next, character a_t is sampled randomly from the distribution p_Θ(a_t| s_{t − 1}). (C) General pipeline of RL system for novel compound generation. (D) Scheme of predictive model. This model takes a SMILES string as an input and provides one real number, which is an estimated property value, as an output. Parameters of the model are trained by l₂-squared loss function minimization.