Some Comments on Shier’s Paper for Inverting Sparse Matrices

Abstract

A paper by Shier (J. Res NBS 80B) shows how to partition the graph of a matrix into a tree so as to minimize the number of operations required to invert the matrix. The present paper shows how to economically solve a sparse system of linear equations after the application of Shier’s method to the coefficient matrix.

1. Introduction

In [1]¹ Shier points out that if: (a) the graph corresponding to a sparse matrix A is partitioned into subgraphs which themselves can be regarded as nodes of a tree, and (b) the nodes of this tree are suitably numbered; then A can be partitioned as (A_ij) where A_ij are submatrices and the ith row of A is

$${\underset{˜}{A}}_i=\left(A_{i1},\cdot \cdot ,A_{ii},0,0,\cdots ,0,A_{i,r(i)},0,\cdots ,0\right),i=1,\cdots ,n$$ (1)

and where node r(i) is the “father” of i in the tree. Also, A_ik = 0 (k < i) unless r(k) = i and A is block incidence-symmetric. He then describes a relatively efficient way of finding A⁻¹, involving the computation of A_ii⁻¹ and similar sub-matrices by standard methods for dense matrices combined with recursive application of his algorithm. He also describes a method for carrying out the tree partitioning in (a) above.

Unfortunately he does not describe in detail how his method can be applied to the much more common problem of solving sparse equations, although he does mention (p. 252, lines 3–5) that it can be so applied. This will now be done.

2. Solution of Equations

We have to solve:

$$A\underset{˜}{x}=\underset{˜}{b}$$ (2)

where A is partitioned as in (1) and x and b are partitioned conformably into sub-vectors

$$({\underset{˜}{x}}_i)\text{\hspace{0.17em}and\hspace{0.17em}}({\underset{˜}{b}}_i)$$ (3)

We may very efficiently solve (2) by Block Gaussian Elimination as follows: (A) (Elimination of sub-diagonal sub-matrices) For i = 1, ⋯, n − 1 do:

$$(\text{I})\text{\hspace{0.17em}Form\hspace{0.17em}multipliers\hspace{0.17em}\hspace{0.17em}}{m}_{r(i),i}=A_{r(i),i}{A}_{ii}{}^{-1}$$ (4)

Eliminate A_r(i),i by subtracting m_r(i),ix (row i) from row r(i), i.e.

$$(\text{II})\text{\hspace{0.17em}Update\hspace{0.17em}}{A}_{r(i),r(i)}\leftarrow A_{r(i),r(i)}-A_{r(i),i}{A}_{ii}{}^{-1}{A}_{i,r(i)}$$ (5)

$$(\text{III})\text{\hspace{0.17em}Update\hspace{0.17em}}{\underset{˜}{b}}_{r(i)}\leftarrow {\underset{˜}{b}}_{r(i)}-A_{r(i),i}{A}_{ii}{}^{-1}{\underset{˜}{b}}_i$$ (6)

(B) (Back-Substitution)

$$(\text{IV}){\underset{˜}{x}}_n=A_{nn}{}^{-1}{\underset{˜}{b}}_n$$ (7)

(V) For i = n − 1, ··, 1 do:

$${\underset{˜}{x}}_i=A_{ii}{}^{-1}[{\underset{˜}{b}}_i-A_{i,r(i)}{\underset{˜}{x}}_{r(i)}]$$ (8)

(The above simply constitutes Gaussian Elimination with coefficients consisting of submatrices instead of scalars.) The great advantage of this method is that there is no fill-in except within the blocks, i.e., a zero submatrix always remains zero.

3. A More Economical Method

Further economy can be obtained by omitting the explicit calculation of m in (4). Rather we can perform triangular decomposition

$$A_{ii}=L_{ii}{U}_{ii}$$ (9)

Then (6) can be replaced by:

$$(\text{I})\text{\hspace{0.17em}Solve\hspace{0.17em}}{L}_{ii}{\underset{˜}{v}}_i={\underset{˜}{b}}_i$$ (10)

$$(\text{II})\text{\hspace{0.17em}Solve\hspace{0.17em}}{U}_{ii}{\underset{˜}{z}}_i={\underset{˜}{v}}_i$$ (11)

$$\text{Then\hspace{0.17em}}{\underset{˜}{z}}_i=A_{ii}{}^{-1}{\underset{˜}{b}}_i$$ (12)

$$(\text{III})\text{\hspace{0.17em}Form\hspace{0.17em}}{\underset{˜}{w}}_{r(i)}=A_{r(i),i}{\underset{˜}{z}}_i$$ (13)

$$(\text{IV}){\underset{˜}{b}}_{r(i)}\leftarrow {\underset{˜}{b}}_{r(i)}-{\underset{˜}{w}}_{r(i)}$$ (14)

(5) Can be replaced by similar calculations with each column of A_i,r(i) taking the place, in turn, of ${\underset{˜}{b}}_i$. (7) Can be replaced by (9), (10), (11) with i = n. (8) Can be replaced by:

$$(\text{I}){\underset{˜}{b}}_i\leftarrow {\underset{˜}{b}}_i-A_{i,r(i)}{\underset{˜}{x}}_{r(i)}$$ (15)

$$(\text{II})\text{\hspace{0.17em}Solve\hspace{0.17em}}{L}_{ii}{\underset{˜}{v}}_i={\underset{˜}{b}}_i$$ (16)

$$(\text{II})\text{\hspace{0.17em}Solve\hspace{0.17em}}{U}_{ii}{\underset{˜}{x}}_i={(\underset{˜}{v})}_i$$ (17)

4. Operation Count

(1) If the explicit inverse A_ii⁻¹ and m_r(i),i are employed as in §2, using dense matrix techniques, the operation count would be as follows, assuming A_ii is order p_ixp_i, and p_i = p for all i: the formation of A_ii⁻¹, m_r(i),i and equation (5) each require 0(p³) multiplications, for a total of 0(3p³); while eq (6) requires 0(p²), (7) requires 0(p³ + p²) and (8) requires 0(2p²). Thus, the total number of multiplications is approximately

$$(n-1)3p^3+(n-1)3p^2+p^3+p^2=(3n-2)(p^3+p^2).$$ (18)

(II) If the method of §3 is used we have: equation (9) requires 0(p³/3) multiplications; equations (10), (11) and (13) together need 0(2p²). The solution of equations (10), (11) and (13) with any column of A_i,r(i) in place of ${\underset{˜}{b}}_i$ requires 0(2p²) for each column, i.e. 0(2p³) in all. Equations (15)–(17) require 0(2p²). Equations (9), (10) and (11) for i = n require $0\left(\frac{p^3}{3}+p^2\right)$. Thus, the total number of multiplications required for this method is approximately

$$\left(\frac{7}{2}n-2\right)p^3+(4n-3)p^2$$ (19)

Thus the method of §3 is more efficient for large p_i = p, when we may ignore multiplicative and overhead factors.

(III) If the equations are solved directly without any partitioning, as if they were full, the number of multiplications required is $\approx \frac{1}{3}{(np)}^3+{(np)}^2$, which for large n is much greater than $\frac{7}{3}n{p}^3$.

5. Labelling of Tree Nodes

The nodes of the tree must be numbered in such a way that its incidence matrix has the form (1). This can be accomplished for example by a modification of the “Reverse Cuthill-McKee Algorithm” [2]. (This was originally devised as a band-width minimization technique, although that aspect has no relevance in the present context.) Simplified and re-worded for our purposes the algorithm may be described thus:

1. Suppose there are N nodes in the tree. Choose an arbitrary node and number it N (this is defined as the only member of “level” 1). Set I = 1 and J = N − 1.

2. Consider all nodes adjacent to nodes in level I but as yet unnumbered (they will be defined as members of level I + 1). Suppose there are K such nodes in all. If K = 0 terminate. Otherwise assign to them the numbers J, J − 1, ··, J − K + 1.

   Set J = J − K and I = I + 1.

   Repeat step B until K = 0.

It is simple to prove that the incidence matrix of a tree thus numbered has the form (1), i.e. each node (numbered i, say) is adjacent to only one node having a higher number (say r(i)).

Proof: Suppose if possible a node numbered i is adjacent to two nodes numbered r₁ and r₂ such that r₁ > i, r₂ > i. Then the nodes r₁ and r₂ belong to lower levels than node i. Hence they are both connected, via paths not including node i, to node N. Thus we have two separate paths connecting nodes i and N, i.e. we have a loop. But this contradicts the assumption that the graph is a tree. Hence there must be only one node adjacent to i with number > i. Q.E.D.