Graphics Processing Units and High-Dimensional Optimization

Abstract

This paper discusses the potential of graphics processing units (GPUs) in high-dimensional optimization problems. A single GPU card with hundreds of arithmetic cores can be inserted in a personal computer and dramatically accelerates many statistical algorithms. To exploit these devices fully, optimization algorithms should reduce to multiple parallel tasks, each accessing a limited amount of data. These criteria favor EM and MM algorithms that separate parameters and data. To a lesser extent block relaxation and coordinate descent and ascent also qualify. We demonstrate the utility of GPUs in nonnegative matrix factorization, PET image reconstruction, and multidimensional scaling. Speedups of 100 fold can easily be attained. Over the next decade, GPUs will fundamentally alter the landscape of computational statistics. It is time for more statisticians to get on-board.

1. INTRODUCTION

Statisticians, like all scientists, are acutely aware that the clock speeds on their desktops and laptops have stalled. Does this mean that statistical computing has hit a wall? The answer fortunately is no, but the hardware advances that we routinely expect have taken an interesting detour. Most computers now sold have two to eight processing cores. Think of these as separate CPUs on the same chip. Naive programmers rely on sequential algorithms and often fail to take advantage of more than a single core. Sophisticated programmers eagerly exploit parallel programming. However, multicore CPUs do not represent the only road to the success of statistical computing.

Graphics processing units (GPUs) have caught the scientific community by surprise. These devices are designed for graphics rendering in computer animation and games. Propelled by these nonscientific markets, the old technology of numerical (array) coprocessors has advanced rapidly. Highly parallel GPUs are now making computational inroads against traditional CPUs in image processing, protein folding, stock options pricing, robotics, oil exploration, data mining, and many other areas (28). We are starting to see orders of magnitude improvement on some hard computational problems. Three companies, Intel, NVIDIA, and AMD/ATI, dominate the market. Intel is struggling to keep up with its more nimble competitors.

Modern GPUs support more vector and matrix operations, stream data faster, and possess more local memory per core than their predecessors. They are also readily available as commodity items that can be inserted as video cards on modern PCs. GPUs have been criticized for their hostile programming environment and lack of double precision arithmetic and error correction, but these faults are being rectified. The CUDA programming environment (27) for NVIDIA chips is now easing some of the programming chores. We could say more about near-term improvements, but most pronouncements would be obsolete within months.

Oddly, statisticians have been slow to embrace the new technology. Silberstein et al (31) first demonstrated the potential for GPUs in fitting simple Bayesian networks. Recently Suchard and Rambaut (33) have seen greater than 100-fold speed-ups in MCMC simulations in molecular phylogeny. Lee et al (18) and Tib-bits, et al (36) are following suit with Bayesian model fitting via particle filtering and slice sampling. Finally, work is under-way to port common data mining techniques such as hierarchical clustering and multi-factor dimensionality reduction onto GPUs (32). These efforts constitute the first wave of an eventual flood of statistical and data mining applications. The porting of GPU tools into the R environment will undoubtedly accelerate the trend (3).

Not all problems in computational statistics can benefit from GPUs. Sequential algorithms are resistant unless they can be broken into parallel pieces. For example, least squares and singular value decomposition–two tasks frequently performed in statistics–cannot benefit from GPUs unless they are extremely large scale or many such small problems need to be solved simultaneously. Even parallel algorithms can be problematic if the entire range of data must be accessed by each GPU. A case in point is the alternating least squares strategy for the nonnegative matrix factorization problem featured in Section 3.1. Because they have limited memory, GPUs are designed to operate on short streams of data. The greatest speedups occur when all of the GPUs on a card perform the same arithmetic operation simultaneously. Effective applications of GPUs in optimization involves both separation of data and separation of parameters.

In the current paper, we illustrate how GPUs can work hand in glove with the MM algorithm, a generalization of the EM algorithm. In many optimization problems, the MM algorithm explicitly separates parameters by replacing the objective function by a sum of surrogate functions, each of which involves a single parameter. Optimization of the one-dimensional surrogates can be accomplished by assigning each subproblem to a different core. Provided the different cores each access just a slice of the data, the parallel subproblems execute quickly. By construction the new point in parameter space improves the value of the objective function. In other words, MM algorithms are iterative ascent or descent algorithms. If they are well designed, then they separate parameters in high-dimensional problems. This is where GPUs enter. They offer most of the benefits of distributed computer clusters at a fraction of the cost. For this reason alone, computational statisticians need to pay attention to GPUs.

Before formally defining the MM algorithm, it may help the reader to walk through a simple numerical example stripped of statistical content. Consider the Rosenbrock test function

$$\begin{array}{l}f(\mathbf{x})=100{(x_1^2-x_2)}^2+{(x_1-1)}^2\\ =100(x_1^4+x_2^2-2x_1^2{x}_2)+(x_1^2-2x_1+1),\end{array}$$ (1.1)

familiar from the minimization literature. As we iterate toward the minimum at x = 1 = (1, 1), we construct a surrogate function that separates parameters. This is done by exploiting the obvious majorization

$$-2x_1^2{x}_2\le x_1^4+x_2^2+{(x_{n1}^2+x_{n2})}^2-2(x_{n1}^2+x_{n2})(x_1^2+x_2),$$

where equality holds when x and the current iterate x_n coincide. It follows that f(x) itself is majorized by the sum of the two surrogates

$$\begin{array}{l}{g}_1(x_1\mid {\mathbf{x}}_n)=200x_1^4-[200(x_{n1}^2+x_{n2})-1]x_1^2-2x_1+1\\ g_2(x_2\mid {\mathbf{x}}_n)=200x_2^2-200(x_{n1}^2+x_{n2})x_2+{(x_{n1}^2+x_{n2})}^2.\end{array}$$

The left panel of Figure 1 depicts the Rosenbrock function and its majorization g₁(x₁ | x_n) + g₂(x₂ | x_n) at the point −1.

Fig 1.

Left: The Rosenbrock (banana) function (the lower surface) and a majorization function at point (−1, −1) (the upper surface). Right: MM iterates.

According to the MM recipe, at each iteration one must minimize the quartic polynomial g₁(x₁ | x_n) and the quadratic polynomial g₂(x₂ | x_n). The quartic possesses either a single global minimum or two local minima separated by a local maximum These minima are the roots of the cubic function $g_1^{\prime }(x_1\mid {\mathbf{x}}_n)$ and can be explicitly computed. We update x₁ by the root corresponding to the global minimum and x₂ via $x_{n+1,2}=\frac{1}{2}(x_{n1}^2+x_{n2})$. The right panel of Figure 1 displays the iterates starting from x₀ = −1. These immediately jump into the Rosenbrock valley and then slowly descend to 1.

Separation of parameters in this example makes it easy to decrease the objective function. This almost trivial advantage is amplified when we optimize functions depending on tens of thousands to millions of parameters. In these settings, Newton’s method and variants such as Fisher’s scoring are fatally handicapped by the need to store, compute, and invert huge Hessian or information matrices. On the negative side of the balance sheet, MM algorithms are often slow to converge. This disadvantage is usually outweighed by the speed of their updates even in sequential mode. If one can harness the power of parallel processing GPUs, then MM algorithms become the method of choice for many high-dimensional problems.

We conclude this introduction by sketching a roadmap to the rest of the paper. Section 2 reviews the MM algorithm. Section 3 discusses three high-dimensional MM examples. Although the algorithm in each case is known, we present brief derivations to illustrate how simple inequalities drive separation of parameters. We then implement each algorithm on a realistic problem and compare running times in sequential and parallel modes. We purposefully omit programming syntax since many tutorials already exist for this purpose, and material of this sort is bound to be ephemeral. The two tutorials (34; 35) are a good place to start for statisticians. Section 4 concludes with a brief discussion of other statistical applications of GPUs and other methods of accelerating optimization algorithms.

2. MM ALGORITHMS

The MM algorithm like the EM algorithm is a principle for creating optimization algorithms. In minimization the acronym MM stands for majorization-minimization; in maximization it stands for minorization-maximization. Both versions are convenient in statistics. For the moment we will concentrate on maximization.

Let f(θ) be the objective function whose maximum we seek. Its argument θ can be high-dimensional and vary over a constrained subset Θ of Euclidean space. An MM algorithm involves minorizing f(θ) by a surrogate function g(θ | θ_n) anchored at the current iterate θ_n of the search. The subscript n indicates iteration number throughout this article. If θ_n+1 denotes the maximum of g(θ | θ_n) with respect to its left argument, then the MM principle declares that θ_n+1 increases f(θ) as well. Thus, MM algorithms revolve around a basic ascent property.

Minorization is defined by the two properties

$$f({\mathit{\theta }}_n)=g({\mathit{\theta }}_n\mid {\mathit{\theta }}_n)$$ (2.1)

$$f(\mathit{\theta })\ge g(\mathit{\theta }\mid {\mathit{\theta }}_n),\mathit{\theta }\ne {\mathit{\theta }}_n.$$ (2.2)

In other words, the surface θ ↦ g(θ | θ_n) lies below the surface θ ↦ f (θ) and is tangent to it at the point θ = θ_n. Construction of the minorizing function g(θ | θ_n) constitutes the first M of the MM algorithm. In our examples g(θ | θ_n) is chosen to separate parameters.

In the second M of the MM algorithm, one maximizes the surrogate g(θ | θ_n) rather than f(θ) directly. It is straightforward to show that the maximum point θ_n+1 satisfies the ascent property f (θ_n+1) ≥ f (θ_n). The proof

$$f({\mathit{\theta }}_{n+1})\ge g({\mathit{\theta }}_{n+1}\mid {\mathit{\theta }}_n)\ge g({\mathit{\theta }}_n\mid {\mathit{\theta }}_n)=f({\mathit{\theta }}_n)$$

reflects definitions (2.1) and (2.2) and the choice of θ_n+1. The ascent property is the source of the MM algorithm’s numerical stability and remains valid if we merely increase g(θ | θ_n) rather than maximize it. In many problems MM updates are delightfully simple to code, intuitively compelling, and automatically consistent with parameter constraints. In minimization we seek a majorizing function g(θ | θ_n) lying above the surface θ ↦ f (θ) and tangent to it at the point θ = θ_n. Minimizing g(θ | θ_n) drives f(θ) downhill.

The celebrated Expectation-Maximization (EM) algorithm (8; 22) is a special case of the MM algorithm. The Q-function produced in the E step of the EM algorithm constitutes a minorizing function of the loglikelihood. Thus, both EM and MM share the same advantages: simplicity, stability, graceful adaptation to constraints, and the tendency to avoid large matrix inversion. The more general MM perspective frees algorithm derivation from the missing data straitjacket and invites wider applications. For example, our multi-dimensional scaling (MDS) and non-negative matrix factorization (NNFM) examples involve no likelihood functions. Wu and Lange (40) briefly summarize the history of the MM algorithm and its relationship to the EM algorithm.

The convergence properties of MM algorithms are well-known (16). In particular, five properties of the objective function f(θ) and the MM algorithm map θ ↦ M(θ) guarantee convergence to a stationary point of f(θ): (a) f(θ) is coercive on its open domain; (b) f(θ) has only isolated stationary points; (c) M(θ) is continuous; (d) θ^* is a fixed point of M(θ) if and only if θ^* is a stationary point of f(θ); and (e) f[M(θ^*)] ≥ f(θ^*), with equality if and only if θ^* is a fixed point of M(θ). These conditions are easy to verify in many applications. The local rate of convergence of an MM algorithm is intimately tied to how well the surrogate function g(θ | θ^*) approximates the objective function f(θ) near the optimal point θ^*.

3. NUMERICAL EXAMPLES

In this section, we compare the performances of the CPU and GPU implementations of three classical MM algorithms coded in C++: (a) non-negative matrix factorization (NNMF), (b) positron emission tomography (PET), and (c) multidimensional scaling (MDS). In each case we briefly derive the algorithm from the MM perspective. For the CPU version, we iterate until the relative change

$$\frac{\mid f({\mathit{\theta }}_n)-f({\mathit{\theta }}_{n-1})\mid }{\mid f({\mathit{\theta }}_{n-1})\mid +1}$$

of the objective function f(θ) between successive iterations falls below a pre-set threshold ε or the number of iterations reaches a pre-set number n_max, whichever comes first. In these examples, we take ε = 10⁻⁹ and n_max = 100, 000. For ease of comparison, we iterate the GPU version for the same number of steps as the CPU version. Overall, we see anywhere from a 22-fold to 112-fold decrease in total run time. The source code is freely available from the first author.

Table 1 shows how our desktop system is configured. Although the CPU is a high-end processor with four cores, we use just one of these for ease of comparison. In practice, it takes considerable effort to load balance the various algorithms across multiple CPU cores. With 240 GPU cores, the GTX 280 GPU card delivers a peak performance of about 933 GFlops in single precision. This card is already obsolete. Newer cards possess twice as many cores, and up to four cards can fit inside a single desktop computer. It is relatively straightforward to program multiple GPUs. Because previous generation GPU hardware is largely limited to single precision, this is a worry in scientific computing. To assess the extent of roundoff error, we display the converged values of the objective functions to ten significant digits. Only rarely is the GPU value far off the CPU mark. The extra effort in programming the GPU version is relatively light. Exploiting the standard CUDA library (27), it takes 77, 176, and 163 extra lines of GPU code to implement the NNMF, PET, and MDS examples, respectively. Finally, for the PET and MDS examples, we also list run times of a CPU implementation with quasi-Newton acceleration method (42). This generic acceleration significantly reduces the number of MM iterations until convergence.

Table 1.

Configuration of the desktop system

	CPU	GPU
Model	Intel Core 2	NVIDIA GeForce
	Extreme X9440	GTX 280
# Cores	4	240
Clock	3.2G	1.3G
Memory	16G	1G

3.1 Non-Negative Matrix Factorizations

Non-negative matrix factorization (NNMF) is an alternative to principle component analysis useful in modeling, compressing, and interpreting nonnegative data such as observational counts and images. The articles (19; 20; 2) discuss in detail algorithm development and statistical applications of NNMF. The basic problem is to approximate a data matrix X with nonnegative entries x_ij by a product VW of two low rank matrices V and W with nonnegative entries v_ik and w_kj. Here X, V, and W are p × q, p × r, and r ×q, respectively, with r much smaller than min{p, q}. One version of NNMF minimizes the objective function

$$f(\mathbf{V},\mathbf{W})={\left|\right|\mathbf{X}-\mathbf{VW}\left|\right|}_{\text{F}}^2=\sum _i\sum _j{\left(x_{ij}-\sum _k{v}_{ik}{w}_{kj}\right)}^2,$$ (3.1)

where || · ||_F denotes the Frobenius-norm. To get an idea of the scale of NNFM imaging problems, p (number of images) can range 10¹–10⁴, q (number of pixels per image) can surpass 10²–10⁴, and one seeks a rank r approximation of about 50. Notably, part of the winning solution of the Netflix challenge relies on variations of NNMF (13). For the Netflix data matrix, p = 480, 000 (raters), q = 18, 000 (movies), and r ranged from 20 to 100.

Exploiting the convexity of the function x ↦ (x_ij − x)², one can derive the inequality

$${\left(x_{ij}-\sum _k{v}_{ik}{w}_{kj}\right)}^2\le \sum _k\frac{a_{\mathit{nikj}}}{b_{\mathit{nij}}}{\left(x_{ij}-\frac{b_{\mathit{nij}}}{a_{\mathit{nikj}}}{v}_{ik}{w}_{kj}\right)}^2$$

where a_nikj = v_nikw_nkj and b_nij = Σ_k a_nikj. This leads to the surrogate function

$$g(\mathbf{V},\mathbf{W}\mid {\mathbf{V}}_n,{\mathbf{W}}_n)=\sum _i\sum _j\sum _k\frac{a_{\mathit{nikj}}}{b_{\mathit{nij}}}{\left(x_{ij}-\frac{b_{\mathit{nij}}}{a_{\mathit{nikj}}}{v}_{ik}{w}_{kj}\right)}^2$$ (3.2)

majorizing the objective function $f(\mathbf{V},\mathbf{W})={\left|\right|\mathbf{X}-\mathbf{VW}\left|\right|}_{\text{F}}^2$. Although the majorization (3.2) does not achieve a complete separation of parameters, it does if we fix V and update W or vice versa. This strategy is called block relaxation (7).

If we elect to minimize g(V, W | V_n, W_n) holding W fixed at W_n, then the stationarity condition for V reads

$$\frac{\partial }{\partial v_{ik}}g(\mathbf{V},{\mathbf{W}}_n\mid {\mathbf{V}}_n,{\mathbf{W}}_n)=-2\sum _j\left(x_{ij}-\frac{b_{\mathit{nij}}}{a_{\mathit{nikj}}}{v}_{ik}{w}_{\mathit{nkj}}\right)w_{\mathit{nkj}}=0.$$

Its solution furnishes the simple multiplicative update

$$v_{n+1,ik}=v_{\mathit{nik}}\frac{{\sum }_j{x}_{ij}{w}_{\mathit{nkj}}}{{\sum }_j{b}_{\mathit{nij}}{w}_{\mathit{nkj}}}.$$ (3.3)

Likewise the stationary condition

$$\frac{\partial }{\partial w_{kj}}g({\mathbf{V}}_{n+1},\mathbf{W}\mid {\mathbf{V}}_{n+1},{\mathbf{W}}_n)=0$$

gives the multiplicative update

$$w_{n+1,kj}=w_{\mathit{nkj}}\frac{{\sum }_i{x}_{ij}{v}_{n+1,ik}}{{\sum }_i{c}_{\mathit{nij}}{v}_{n+1,ik}},$$ (3.4)

where c_nij = Σ_k v_n+1,ikw_nkj. Close inspection of the multiplicative updates (3.3) and (3.4) shows that their numerators depend on the matrix products $\mathbf{X}{\mathbf{W}}_n^t$ and ${\mathbf{V}}_{n+1}^t\mathbf{X}$ and their denominators depend on the matrix products ${\mathbf{V}}_n{\mathbf{W}}_n{\mathbf{W}}_n^t$ and ${\mathbf{V}}_{n+1}^t{\mathbf{V}}_{n+1}{\mathbf{W}}_n$. Large matrix multiplications are very fast on GPUs because CUDA implements in parallel the BLAS (basic linear algebra subprograms) library widely applied in numerical analysis (26). Once the relevant matrix products are available, each elementwise update of v_ik or w_kj involves just a single multiplication and division. These scalar operations are performed in parallel through hand-written GPU code. Algorithm 1 summarizes the steps in performing NNMF.

Algorithm 1.

(NNMF) Given $\mathbf{X}\in {\mathbb{R}}_{+}^{p\times q}$, find $\mathbf{V}\in {\mathbb{R}}_{+}^{p\times r}$ and $\mathbf{W}\in {\mathbb{R}}_{+}^{r\times q}$ minimizing ${\left|\right|\mathbf{X}-\mathbf{VW}\left|\right|}_{\text{F}}^2$.

Initialize: Draw v0ik and w0kj uniform on (0,1) for all 1 ≤ i ≤ p, 1 ≤ k ≤ r, 1 ≤ j ≤ q

repeat

Compute $\mathbf{X}{\mathbf{W}}_n^t$ and ${\mathbf{V}}_n{\mathbf{W}}_n{\mathbf{W}}_n^t$

$v_{n+1,ik}\leftarrow v_{\mathit{nik}}·{\{\mathbf{X}{\mathbf{W}}_n^t\}}_{ik}/{\{{\mathbf{V}}_n{\mathbf{W}}_n{\mathbf{W}}_n^t\}}_{ik}$ for all 1 ≤ i ≤ p, 1 ≤ k ≤ r

Compute ${\mathbf{V}}_{n+1}^t\mathbf{X}$ and ${\mathbf{V}}_{n+1}^t{\mathbf{V}}_{n+1}{\mathbf{W}}_n$

$w_{n+1,kj}\leftarrow w_{\mathit{nkj}}·{\{{\mathbf{V}}_{n+1}^t\mathbf{X}\}}_{kj}/{\{{\mathbf{V}}_{n+1}^t{\mathbf{V}}_{n+1}{\mathbf{W}}_n\}}_{kj}$ for all 1 ≤ k ≤ r, 1 ≤ j ≤ q

until convergence occurs

We now compare CPU and GPU versions of the multiplicative NNMF algorithm on a training set of face images. Database #1 from the MIT Center for Biological and Computational Learning (CBCL) (25) reduces to a matrix X containing p = 2, 429 gray scale face images with q = 19 × 19 = 361 pixels per face. Each image (row) is scaled to have mean and standard deviation 0.25. Figure 2 shows the recovery of the first face in the database using a rank r = 49 decomposition. The 49 basis images (rows of W) represent different aspects of a face. The rows of V contain the coefficients of these parts estimated for the various faces. Some of these facial features are immediately obvious in the reconstruction. Table 2 compares the run times of Algorithm 1 implemented on our CPU and GPU respectively. We observe a 22 to 112-fold speed-up in the GPU implementation. Run times for the GPU version depend primarily on the number of iterations to convergence and very little on the rank r of the approximation. Run times of the CPU version scale linearly in both the number of iterations and r.

Fig 2.

Approximation of a face image by rank-49 NNMF: coefficients × basis images = approximate image.

Table 2.

Run-time (in seconds) comparisons for NNMF on the MIT CBCL face image data. The dataset contains p = 2, 429 faces with q = 19 × 19 = 361 pixels per face. The columns labeled Function refer to the converged value of the objective function.

Rank r	Iters	CPU		GPU		Speedup
		Time	Function	Time	Function
10	25459	1203	106.2653503	55	106.2653504	22
20	87801	7564	89.56601262	163	89.56601287	46
30	55783	7013	78.42143486	103	78.42143507	68
40	47775	7880	70.05415929	119	70.05415950	66
50	53523	11108	63.51429261	121	63.51429219	92
60	77321	19407	58.24854375	174	58.24854336	112

It is worth stressing a few points. First, the objective function (3.1) is convex in V for W fixed, and vice versa but not jointly convex. Thus, even though the MM algorithm enjoys the descent property, it is not guaranteed to find the global minimum (2). There are two good alternatives to the multiplicative algorithm. First, pure block relaxation can be conducted by alternating least squares (ALS). In updating V with W fixed, ALS omits majorization and solves the p separated nonnegative least square problems

$$\underset{\mathbf{V}(i,:)}{min}{\left|\right|\mathbf{X}(i,:)-\mathbf{V}(i,:)\mathbf{W}]||}_2^2\text{subject}\text{to}\mathbf{V}(i,:)\ge 0,$$

where V(i, :) and X(i, :) denote the i-th row of the corresponding matrices. Similarly, in updating W with V fixed, ALS solves q separated nonnegative least square problems. Separation naturally suggests parallel implementations; but parallelization by GPUs hits a snag because each nonnegative least square subproblem needs to operate on the whole W matrix. Another possibility is to change the objective function to

$$L(\mathbf{V},\mathbf{W})=\sum _i\sum _j\left[x_{ij}ln\left(\sum _k{v}_{ik}{w}_{kj}\right)-\sum _k{v}_{ik}{w}_{kj}\right]$$

according to a Poisson model for the counts x_ij (19). This works even when some entries x_ij fail to be integers, but the Poisson loglikelihood interpretation is lost. A pure MM algorithm for maximizing L(V, W) is

$$v_{n+1,ik}=v_{\mathit{nik}}\sqrt{\frac{{\sum }_j{x}_{ij}{w}_{\mathit{nkj}}/b_{\mathit{nij}}}{{\sum }_j{w}_{\mathit{nkj}}}},w_{n+1,ij}=w_{\mathit{nkj}}\sqrt{\frac{{\sum }_i{x}_{ij}{v}_{\mathit{nik}}/b_{\mathit{nij}}}{{\sum }_i{v}_{\mathit{nik}}}}.$$

Derivation of these variants of Lee and Seung’s (19) Poisson updates is left to the reader.

3.2 Positron Emission Tomography

The field of computed tomography has exploited EM algorithms for many years. In positron emission tomography (PET), the reconstruction problem consists of estimating the Poisson emission intensities λ = (λ₁, …, λ_p) of p pixels arranged in a 2-dimensional grid surrounded by an array of photon detectors. The observed data are coincidence counts (y₁, … y_d) along d lines of flight connecting pairs of photon detectors. The loglikelihood under the PET model is

$$L(\mathbf{\lambda })=\sum _i\left[y_{i}ln\left(\sum _j{e}_{ij}{\lambda }_j\right)-\sum _j{e}_{ij}{\lambda }_j\right],$$

where the e_ij are constants derived from the geometry of the grid and the detectors. Without loss of generality, one can assume Σ_i e_ij = 1 for each j. It is straightforward to derive the traditional EM algorithm (14; 39) from the MM perspective using the concavity of the function ln s. Indeed, application of Jensen’s inequality produces the minorization

$$L(\mathbf{\lambda })\ge \sum _i{y}_i\sum _j{w}_{\mathit{nij}}ln\left(\frac{e_{ij}{\lambda }_j}{w_{\mathit{nij}}}\right)-\sum _i\sum _j{e}_{ij}{\lambda }_j=Q(\mathbf{\lambda }\mid {\mathbf{\lambda }}_n),$$

where w_nij = e_ijλ_nj/(Σ_k e_ikλ_nk). This maneuver again separates parameters. The stationarity conditions for the surrogate Q(λ | λ _n) supply the parallel updates

$${\lambda }_{n+1,j}=\frac{{\sum }_i{y}_i{w}_{\mathit{nij}}}{{\sum }_i{e}_{ij}}.$$ (3.5)

The convergence of the PET algorithm (3.5) is frustratingly slow, even under systematic acceleration (30; 42). Furthermore, the reconstructed images are of poor quality with a grainy appearance. The early remedy of premature halting of the algorithm cuts computational cost but is entirely ad hoc, and the final image depends on initial conditions. A better option is add a roughness penalty to the loglikelihood. This device not only produces better images but also accelerates convergence. Thus, we maximize the penalized loglikelihood

$$f(\mathbf{\lambda })=L(\mathbf{\lambda })-\frac{\mu }{2}\sum _{\{j,k\}\in \mathcal{N}}{({\lambda }_j-{\lambda }_k)}^2$$ (3.6)

where μ is the roughness penalty constant, and is the neighborhood system that pairs spatially adjacent pixels. An absolute value penalty is less likely to deter the formation of edges than a square penalty, but it is easier to deal with a square penalty analytically, and we adopt it for the sake of simplicity. In practice, visual inspection of the recovered images guides the selection of the roughness penalty constant μ.

To maximize f(λ) by an MM algorithm, we must minorize the penalty in a manner consistent with the separation of parameters. In view of the evenness and convexity of the function s², we have

$${({\lambda }_j-{\lambda }_k)}^2\le \frac{1}{2}{(2{\lambda }_j-{\lambda }_{nj}-{\lambda }_{nk})}^2+\frac{1}{2}{(2{\lambda }_k-{\lambda }_{nj}-{\lambda }_{nk})}^2.$$

Equality holds if λ_j + λ_k = λ_nj + λ_nk, which is true when λ = λ_n. Combining our two minorizations furnishes the surrogate function

$$g(\mathbf{\lambda }\mid {\mathbf{\lambda }}_n)=Q(\mathbf{\lambda }\mid {\mathbf{\lambda }}_n)-\frac{\mu }{4}\sum _{\{j,k\}\in \mathcal{N}}\left[{(2{\lambda }_j-{\lambda }_{nj}-{\lambda }_{nk})}^2+{(2{\lambda }_k-{\lambda }_{nj}-{\lambda }_{nk})}^2\right].$$

To maximize g(λ | λ_n), we define = {k : {j, k} ∈ } and set the partial derivative

$$\frac{\partial }{\partial {\lambda }_j}g(\mathbf{\lambda }\mid {\mathbf{\lambda }}_n)=\sum _i\left[\frac{y_i{w}_{\mathit{nij}}}{{\lambda }_j}-e_{ij}\right]-\mu \sum _{k:\in {\mathcal{N}}_j}(2{\lambda }_j-{\lambda }_{nj}-{\lambda }_{nk})$$ (3.7)

equal to 0 and solve for λ_n+1,j. Multiplying equation (3.7) by λ_j produces a quadratic with roots of opposite signs. We take the positive root

$${\lambda }_{n+1,j}=\frac{-b_{nj}-\sqrt{b_{nj}^2-4a_j{c}_{nj}}}{2a_j},$$

where

$$a_j=-2\mu \sum _{k\in {\mathcal{N}}_j}1,b_{nj}=\sum _{k\in {\mathcal{N}}_j}({\lambda }_{nj}+{\lambda }_{nk})-1,c_{nj}=\sum _i{y}_i{w}_{\mathit{nij}}.$$

Algorithm 2 summarizes the complete MM scheme. Obviously, complete parameter separation is crucial. The quantities a_j can be computed once and stored. The quantities b_nj and c_nj are computed for each j in parallel. To improve GPU performance in computing the sums over i, we exploit the widely available parallel sum-reduction techniques (31). Given these results, a specialized but simple GPU code computes the updates λ_n+1,j for each j in parallel.

Algorithm 2.

(PET Image Recovering) Given the coefficient matrix $\mathbf{E}\in {\mathbb{R}}_{+}^{d\times p}$, coincident counts $\mathbf{y}=(y_1,\dots ,y_d)\in {\mathbf{Z}}_{+}^d$, and roughness parameter μ > 0, find the intensity vector $\mathbf{\lambda }=({\lambda }_1,\dots ,{\lambda }_p)\in {\mathbb{R}}_{+}^p$ that maximizes the objective function (3.6).

Scale E to have unit l1 column norms.

Compute | | = Σk:{j,k}∈ 1 and aj − 2μ| | for all 1 ≤ j ≤ p.

Initialize: λ0j ← 1, j = 1, …, p.

repeat

znij ← (yieijλnj)/(Σkeikλnk) for all 1 ≤ i ≤ d, 1 ≤ j ≤ p

for j = 1 to p do

bnj ← μ(| | λnj + Σk∈ λnk) − 1

cnj ← Σi znij

${\lambda }_{n+1,j}\leftarrow (-b_{nj}-\sqrt{b_{nj}^2-4a_j{c}_{nj}})/(2a_j)$

end for

until convergence occurs

Table 3 compares the run times of the CPU and GPU implementations for a simulated PET image (30). The image as depicted in the top of Figure 3 has p = 64 × 64 = 4, 096 pixels and is interrogated by d = 2, 016 detectors. Overall we see a 43- to 53-fold reduction in run times with the GPU implementation. Figure 3 displays the true image and the estimated images under penalties of μ = 0, 10⁻⁵, 10⁻⁶, and 10⁻⁷. Without penalty (μ = 0), the algorithm fails to converge in 100,000 iterations.

Table 3.

Comparison of run times (in seconds) for a PET imaging problem on the simulated data in (30). The image has p = 64 × 64 = 4, 096 pixels and is interrogated by d = 2, 016 detectors. The columns labeled Function refer to the converged value of the objective function. The results under the heading QN(10) on CPU invoke quasi-Newton acceleration (42) with 10 secant conditions.

Penalty μ	CPU			GPU				QN(10) on CPU
	Iters	Time	Function	Iters	Time	Function	Speedup	Iters	Time	Function	Speedup
0	100000	14790	−7337.152765	100000	282	−7337.153387	52	6549	2094	−7320.100952	n/a
10−7	24457	3682	−8500.083033	24457	70	−8508.112249	53	251	83	−8500.077057	44
10−6	6294	919	−15432.45496	6294	18	−15432.45586	51	80	29	−15432.45366	32
10−5	589	86	−55767.32966	589	2	−55767.32970	43	19	9	−55767.32731	10

Fig 3.

The true PET image (top) and the recovered images with penalties μ = 0, 10⁻⁷, 10⁻⁶, and 10⁻⁵.

3.3 Multidimensional Scaling

Multidimensional scaling (MDS) was the first statistical application of the MM principle (6; 5). MDS represents q objects as faithfully as possible in p-dimensional space given a nonnegative weight w_ij and a nonnegative dissimilarity measure y_ij for each pair of objects i and j. If θⁱ ∈ ℝ^p is the position of object i, then the p × q parameter matrix θ = (θ¹, …, θ^q) is estimated by minimizing the stress function

$$\begin{array}{l}f(\mathit{\theta })=\sum _{1\le i<j\le q}{w}_{ij}{(y_{ij}-||{\mathit{\theta }}^i-{\mathit{\theta }}^j||)}^2\\ =\sum _{i<j}{w}_{ij}{y}_{ij}^2-2\sum _{i<j}{w}_{ij}{y}_{ij}\left|\right|{\mathit{\theta }}^i-{\mathit{\theta }}^j\left|\right|+\sum _{i<j}{w}_{ij}{\left|\right|{\mathit{\theta }}^i-{\mathit{\theta }}^j\left|\right|}^2,\end{array}$$ (3.8)

where ||θⁱ − θ^j|| is the Euclidean distance between θⁱ and θ^j. The stress function (3.8) is invariant under translations, rotations, and reflections of ℝ^p. To avoid translational and rotational ambiguities, we take θ¹ to be the origin and the first p − 1 coordinates of θ² to be 0. Switching the sign of ${\theta }_p^2$ leaves the stress function invariant. Hence, convergence to one member of a pair of reflected minima immediately determines the other member.

Given these preliminaries, we now review the derivation of the MM algorithm presented in (17). Because we want to minimize the stress, we majorize it. The middle term in the stress (3.8) is majorized by the Cauchy-Schwartz inequality

$$-\left|\right|{\mathit{\theta }}^i-{\mathit{\theta }}^j\left|\right|\le -\frac{{({\mathit{\theta }}^i-{\mathit{\theta }}^j)}^t({\mathit{\theta }}_n^i-{\mathit{\theta }}_n^j)}{\left|\right|{\mathit{\theta }}_n^i-{\mathit{\theta }}_n^j\left|\right|}.$$

To separate the parameters in the summands of the third term of the stress, we invoke the convexity of the Euclidean norm || · || and the square function s². These maneuvers yield

$$\begin{array}{l}{\left|\right|{\mathit{\theta }}^i-{\mathit{\theta }}^j\left|\right|}^2={\Vert \frac{1}{2}\left[2{\mathit{\theta }}^i-({\mathit{\theta }}_n^i+{\mathit{\theta }}_n^j)\right]-\frac{1}{2}\left[2{\mathit{\theta }}^j-({\mathit{\theta }}_n^j+{\mathit{\theta }}_n^j)\right]\Vert }^2\\ \le 2{\Vert {\mathit{\theta }}^i-\frac{1}{2}({\mathit{\theta }}_n^i+{\mathit{\theta }}_n^j)\Vert }^2+2{\Vert {\mathit{\theta }}^j-\frac{1}{2}({\mathit{\theta }}_n^i+{\mathit{\theta }}_n^j)\Vert }^2.\end{array}$$

Assuming that w_ij = w_ji and y_ij = y_ji, the surrogate function therefore becomes

$$\begin{array}{l}g(\mathit{\theta }\mid {\mathit{\theta }}_n)=2\sum _{i<j}{w}_{ij}\left[{\Vert {\mathit{\theta }}^i-\frac{1}{2}({\mathit{\theta }}_n^i+{\mathit{\theta }}_n^j)\Vert }^2-\frac{y_{ij}{({\mathit{\theta }}^i)}^t({\mathit{\theta }}_n^i-{\mathit{\theta }}_n^j)}{\left|\right|{\mathit{\theta }}_n^i-{\mathit{\theta }}_n^j\left|\right|}\right]+2\sum _{i<j}{w}_{ij}\left[{\Vert {\mathit{\theta }}^j-\frac{1}{2}({\mathit{\theta }}_n^i+{\mathit{\theta }}_n^j)\Vert }^2+\frac{y_{ij}{({\mathit{\theta }}^j)}^t({\mathit{\theta }}_n^i-{\mathit{\theta }}_n^j)}{\left|\right|{\mathit{\theta }}_n^i-{\mathit{\theta }}_n^j\left|\right|}\right]\\ =2\sum _{i=1}^q\sum _{j\ne i}\left[w_{ij}{\Vert {\mathit{\theta }}^i-\frac{1}{2}({\mathit{\theta }}_n^i+{\mathit{\theta }}_n^j)\Vert }^2-\frac{w_{ij}{y}_{ij}{({\mathit{\theta }}^i)}^t({\mathit{\theta }}_n^i-{\mathit{\theta }}_n^j)}{\left|\right|{\mathit{\theta }}_n^i-{\mathit{\theta }}_n^j\left|\right|}\right]\end{array}$$

up to an irrelevant constant. Setting the gradient of the surrogate equal to the 0 vector produces the parallel updates

$${\theta }_{n+1,k}^i=\frac{{\sum }_{j\ne i}\left[\frac{w_{ij}{y}_{ij}({\theta }_{nk}^i-{\theta }_{nk}^j)}{\left|\right|{\mathit{\theta }}_n^i-{\mathit{\theta }}_n^j\left|\right|}+w_{ij}({\theta }_{nk}^i+{\theta }_{nk}^j)\right]}{2{\sum }_{j\ne i}{w}_{ij}}$$

for all movable parameters ${\theta }_k^i$.

Algorithm 3 summarizes the parallel organization of the steps. Again the matrix multiplications ${\mathbf{\Theta }}_n^t{\mathbf{\Theta }}_n$ and Θ_n(W − Z_n) can be taken care of by the CUBLAS library (26). The remaining steps of the algorithm are conducted by easily written parallel code.

Algorithm 3.

(MDS) Given weights W and distances Y ∈ ℝ^q×q, find the matrix Θ = [θ¹, …, θ^q] ∈ ℝ^p×q which minimizes the stress (3.8).

Precompute: xij ← wij yij for all 1 ≤ i, j ≤ q

Precompute: wi· ← Σj wij for all 1 ≤ i ≤ q

Initialize: Draw ${\theta }_{0k}^i$ uniformly on [−1,1] for all 1 ≤ i ≤ q, 1 ≤ k ≤ p

repeat

Compute ${\mathbf{\Theta }}_n^t{\mathbf{\Theta }}_n$

$d_{\mathit{nij}}\leftarrow {\{{\mathbf{\Theta }}_n^t{\mathbf{\Theta }}_n\}}_{ii}+{\{{\mathbf{\Theta }}_n^t{\mathbf{\Theta }}_n\}}_{jj}-2{\{{\mathbf{\Theta }}_n^t{\mathbf{\Theta }}_n\}}_{ij}$ for all 1 ≤ i, j ≤ q

znij ← xij/dnij for all 1 ≤ i ≠ j ≤ q

zni· ← Σj znij for all 1 ≤ i ≤ q

Compute Θ;n(W − Zn)

${\theta }_{n+1,k}^i\leftarrow [{\theta }_{nk}^i(w_{i·}+z_{ni·})+{\{{\mathbf{\Theta }}_n(\mathbf{W}-{\mathbf{Z}}_n)\}}_{ik}]/(2w_{i·})$ for all 1 ≤ i ≤ p, 1 ≤ k ≤ q

until convergence occurs

Table 4 compares the run times in seconds for MDS on the 2005 United States House of Representatives roll call votes. The original data consist of the 671 roll calls made by 401 representatives. We refer readers to the reference (9) for a careful description of the data and how the MDS input 401×401 distance matrix is derived. The weights w_ij are taken to be 1. In our notation, the number of objects (House Representatives) is q = 401. Even for this relatively small dataset, we see a 27–48 fold reduction in total run times, depending on the projection dimension p. Figure 4 displays the results in p = 3 dimensional space. The Democratic and Republican members are clearly separated. For p = 30, the algorithm fails to converge within 100,000 iterations.

Table 4.

Comparison of run times (in seconds) for MDS on the 2005 House of Representatives roll call data. The number of points (representatives) is q = 401. The results under the heading QN(20) on CPU invoke the quasi-Newton acceleration (42) with 20 secant conditions.

Dim-p	CPU			GPU				QN(20) on CPU
	Iters	Time	Stress	Iters	Time	Stress	Speedup	Iters	Time	Stress	Speedup
2	3452	43	198.5109307	3452	1	198.5109309	43	530	16	198.5815072	3
3	15912	189	95.55987770	15912	6	95.55987813	32	1124	38	92.82984196	5
4	15965	189	56.83482075	15965	7	56.83482083	27	596	18	56.83478026	11
5	24604	328	39.41268434	24604	10	39.41268444	33	546	17	39.41493536	19
10	29643	441	14.16083986	29643	13	14.16083992	34	848	35	14.16077368	13
20	67130	1288	6.464623901	67130	32	6.464624064	40	810	43	6.464526731	30
30	100000	2456	4.839570118	100000	51	4.839570322	48	844	54	4.839140671	n/a

Fig 4.

Display of the MDS results with p = 3 coordinates on the 2005 House of Representatives roll call data.

Although the projection of points into p > 3 dimensional spaces may sound artificial, there are situations where this is standard practice. First, MDS is foremost a dimension reduction tool, and it is desirable to keep p > 3 to maximize explanatory power. Second, the stress function tends to have multiple local minima in low dimensions (10). A standard optimization algorithm like MM is only guaranteed to converge to a local minima of the stress function. As the number of dimensions increases, most of the inferior modes disappear. One can formally demonstrate that the stress has a unique minimum when p = q − 1 (4; 10). In practice, uniqueness can set in well before p reaches q − 1. In the recent work (41), we propose a “dimension crunching” technique that increases the chance of the MM algorithm converging to the global minimum of the stress function. In dimension crunching, we start optimizing the stress in a Euclidean space ℝ^m with m > p. The last m − p components of each column θⁱ are gradually subjected to stiffer and stiffer penalties. In the limit as the penalty tuning parameter tends to ∞, we recover the global minimum of the stress in ℝ^p. This strategy inevitably incurs a computational burden when m is large, but the MM+GPU combination comes to the rescue.

4. DISCUSSION

The rapid and sustained increases in computing power over the last half century have transformed statistics. Every advance has encouraged statisticians to attack harder and more sophisticated problems. We tend to take the steady march of computational efficiency for granted, but there are limits to a chip’s clock speed, power consumption, and logical complexity. Parallel processing via GPUs is the technological innovation that will power ambitious statistical computing in the coming decade. Once the limits of parallel processing are reached, we may see quantum computers take off. In the meantime statisticians should learn how to harness GPUs productively.

We have argued by example that high-dimensional optimization is driven by parameter and data separation. It takes both to exploit the parallel capabilities of GPUs. Block relaxation and the MM algorithm often generate ideal parallel algorithms. In our opinion the MM algorithm is the more versatile of the two generic strategies. Unfortunately, block relaxation does not accommodate constraints well and may generate sequential rather than parallel updates. Even when its updates are parallel, they may not be data separated. The EM algorithm is one of the most versatile tools in the statistician’s toolbox. The MM principle generalizes the EM algorithm and shares its positive features. Scoring and Newton’s methods become impractical in high dimensions. Despite these arguments in favor of MM algorithms, one should always keep in mind hybrid algorithms such as the one we implemented for NNMF.

Although none of our data sets is really large by today’s standards, they do demonstrate that a good GPU implementation can easily achieve one to two orders of magnitude improvement over a single CPU core. Admittedly, modern CPUs come with 2 to 8 cores, and distributed computing over CPU-based clusters remains an option. But this alternative also carries a hefty price tag. The NVIDIA GTX280 GPU on which our examples were run drives 240 cores at a cost of several hundred dollars. High-end computers with 8 or more CPU nodes cost thousands of dollars. It would take 30 CPUs with 8 cores each to equal a single GPU at the same clock rate. Hence, GPU cards strike an effective and cost efficient balance.

In the three test examples, we implemented and reported performance results on the GTX 280 card in single precision because this particular card is not optimized for double precision computation. As the numerical results show, speedups are gained at loss of accuracy after a certain number of significant digits and in rare cases the single precision calculation on GPU may lead to an inferior mode (e.g., the μ = 10⁻⁷ case in Table 3). In some preliminary experimentation on the same PET imaging algorithm, we found that the performance of a double precision implementation on GTX 280 is about one third of the single precision implementation. In other words, it takes three times longer to perform the same number of iterations. However this lack of double precision support is soon lessened by the rapid advancement in GPU technology. The newer GTX 480 video card has twice as many cores as the GTX 280 and much improved double precision support. On the same desktop system as in Table 1, GTX 480 now delivers 89 fold speedup in single precision and 43 fold speedup in double precision over the CPU code which is in double precision. Not to mention the Tesla C2050 video card, which is already available on market and possesses a peak double precision floating point performance (515 Gflops) that is three times that of GTX 480 (168 Gflops). The challenge for the statistics community is to tackle more and more complicated statistical models on bigger and bigger datasets. Computational statisticians should decide for themselves the speed vs precision trade-off for their problem at hand.

The simplicity of MM algorithms often comes at a price of slow (at best linear) convergence. Our MDS, NNMF, and PET (without penalty) examples are cases in point. Slow convergence is a concern as statisticians head into an era dominated by large data sets and high-dimensional models. Think about the scale of the Netflix data matrix. The speed of any iterative algorithm is determined by both the computational cost per iteration and the number of iterations until convergence. GPU implementation reduces the first cost. Computational statisticians also have a bag of software tricks to decrease the number of iterations (23; 11; 21; 15; 12; 24; 38). For instance, the recent paper (42) proposes a quasi-Newton acceleration scheme particularly suitable for high-dimensional problems. The scheme is off-the-shelf and broadly applies to any search algorithm defined by a smooth algorithm map. The acceleration requires only modest increments in storage and computation per iteration. Tables 3 and 4 also list the results of this quasi-Newton acceleration of the CPU implementation for the MDS and PET examples. As the tables make evident, quasi-Newton acceleration significantly reduces the number of iterations until convergence. The accelerated algorithm always locates a better mode while cutting run times compared to the unaccelerated algorithm. We have tried the quasi-Newton acceleration on our GPU hardware with mixed results. We suspect that the lack of full double precision on the GPU is the culprit. When full double precision becomes widely available, the combination of GPU hardware acceleration and algorithmic software acceleration will be extremely potent.

Successful acceleration methods will also facilitate attacking another nagging problem in computational statistics, namely multimodality. No one knows how often statistical inference is fatally flawed because a standard optimization algorithm converges to an inferior mode. The current remedy of choice is to start a search algorithm from multiple random points. Algorithm acceleration is welcome because the number of starting points can be enlarged without an increase in computing time. As an alternative to multiple starting points, our recent paper (41) suggests modifications of several standard MM algorithms that increase the chance of locating better modes. These simple modifications all involve variations on deterministic annealing (37).

Our treatment of simple classical examples should not hide the wide applicability of the powerful MM+GPU combination. A few other candidate applications include penalized estimation of haplotype frequencies in genetics (1), construction of biological and social networks under a random multigraph model (29), and data mining with a variety of models related to the multinomial distribution (43). Many mixture models will benefit as well from parallelization, particularly in assigning group memberships. Finally, parallelization is hardly limited to optimization. We can expect to see many more GPU applications in MCMC sampling. Given the computationally intensive nature of MCMC, the ultimate payoff may even be higher in the Bayesian setting than in the frequentist setting. For example, in a recent study (35), GPU implementations deliver up to a 140 fold speedup in Bayesian fitting of massive mixture models. Of course realistically, these future triumphs will require a great deal of thought, effort, and education. There is usually a desert to wander and a river to cross before one reaches the promised land.