Algorithm 1. Emergent property inference

1 initialize $\mathit{𝜽}$ by fitting $q_{\mathit{𝜽}}$ to an isotropic gaussian of mean ${\mathit{𝝁}}_{\text{init}}$ and variance ${\mathit{𝝈}}_{\text{init}}^2$ 2 initialize $c_0>0$ and ${\mathit{𝜼}}_{\text{opt},0}=\mathrm{}$. 3 for Augmented lagrangian epoch $k=1,\mathrm{\dots },k_{\text{max}}$ do 4     for SGD iteration $i=1,\mathrm{\dots },i_{\text{max}}$ do 5          Sample ${\mathbf{𝐳}}_0^{(1)},\mathrm{\dots },{\mathbf{𝐳}}_0^{(n)}\sim q_0$, get transformed variable ${\mathbf{𝐳}}^{(j)}=g_{\mathit{𝜽}}({\mathbf{𝐳}}_0^{(j)})$, $j=1,\mathrm{\dots },n$ 6          Update $\mathit{𝜽}$ by descending its stochastic gradient (using ADAM optimizer [Kingma and Ba, 2015]).           ${\displaystyle \begin{array}{ll}{\displaystyle {\mathrm{\nabla }}_{\mathit{\theta }}L(\mathit{\theta };{\mathit{\eta }}_{\text{opt},k},c)}& {\displaystyle =\frac{1}{n}\sum _{j=1}^n{\mathrm{\nabla }}_{\mathit{\theta }}\log q_{\mathit{\theta }}({\mathbf{z}}^{(j)})+\frac{1}{n}\sum _{j=1}^n{\mathrm{\nabla }}_{\mathit{\theta }}\left(T\left({\mathbf{z}}^{(j)}\right)-{\mathit{\mu }}_{\text{opt}}\right){\mathit{\eta }}_{\text{opt},k}}\\ & {\displaystyle +c_k\frac{2}{n}\sum _{j=1}^{\frac{n}{2}}{\mathrm{\nabla }}_{\mathit{\theta }}\left(T\left({\mathbf{z}}^{(j)}\right)-{\mathit{\mu }}_{\text{opt}}\right)\cdot \frac{2}{n}\sum _{j=\frac{n}{2}+1}^n\left(T\left({\mathbf{z}}^{(j)}\right)-{\mathit{\mu }}_{\text{opt}}\right)}\end{array}}$ 7     end 8     Sample ${\mathbf{𝐳}}_0^{(1)},\mathrm{\dots },{\mathbf{𝐳}}_0^{(n)}\sim q_0$, get transformed variable ${\mathbf{𝐳}}^{(j)}=g_{\mathit{𝜽}}({\mathbf{𝐳}}_0^{(j)})$, $j=1,\mathrm{\dots },n$ 9     Update ${\mathit{𝜼}}_{\text{opt},k+1}={\mathit{𝜼}}_{\text{opt},k}+c_k\frac{1}{n}{\sum }_{j=1}^n\left(T\left({\mathbf{𝐳}}^{(j)}\right)-{\mathit{𝝁}}_{\text{opt}}\right)$. 10     Update $c_{k+1}>c_k$ (see text for detail). 11 end