Algorithm 1.

Incremental Ridge Regression with RBF Random Fourier Features.

Require: λ > 0, γ > 0, n > 0, D > 0
1: A−1 ← λ−1 ID × D
2: w ← 0D × 1
3: β ← 0D × 1
4: Ω ~ (0, 2 γ)D × n
5: b ~ (0, 2π)D × 1
6: for all (x, y) do
7:   $z\leftarrow \sqrt{\frac{2}{D}}\cos \text{(}\Omega x+b)$	// random feature mapping
8:   ŷ ← wT z	// predict label
9:   β ← β + zy
10:   $A^{-1}\leftarrow A^{-1}-\frac{A^{-1}z{z}^{\text{T}}{A}^{-1}}{1+z^{\text{T}}{A}^{-1}z}$	// Sherman–Morrison
11:   w ← A−1β	// update weights
12:   yield ŷ
13: end for