Algorithm 1 Stochastic Gradient Descent

1: for $j=1$ to $n$ do

2: Draw $i\in \left\{1,2,\dots ,m\right\}$ at random.

3: ${\theta }_j\leftarrow {\theta }_j-\alpha {\nabla }_{\theta }J\left(x_j^{(i)},{\theta }_j\right)$

4: end for

5: return ${\theta }_j$