Linear Algebra for Graphics Geeks (SVD-X)

This was previously posted. After realizing it was redundant in light of an earlier post, I retracted it. On third thought, I'm reposting as it has additional thoughts on the subject of the SVD and eigen decomposition.

If a matrix is symmetric matrix, then

A = A^T

Expressed in terms of the SVD

USV^T = (USV^T)^T
USV^T = VSU^T

In the case of a symmetric matrix

AA^T = A^TA = A²

Since these are equal and U contains the eigenvectors of AA^T and V contains the eigenvectors of A^TA, U and V must be equal and both contain the eigenvectors of A².

Let's state that again.

In the case of the SVD of symmetric matrix A:

U = V = eigenvectors of A².
S = eigenvalues of A².

And since U and V are equal...

USU^T = VSV^T

Given that realization, the next question crossing my mind was what happens if you calculate the SVD of A^TA?

Considering the previous conclusions and A^TA being a symmetric matrix, it follows that:

U = V = eigenvectors of (A^TA)²
S = eigenvalues of (A^TA)²

This led me to consider the relationship between the eigenvectors of A^TA and the eigenvectors of (A^TA)².

Consider the eigen decomposition of A^TA

A^TA = USU^T

Square both sides...

(A^TA)² = (USU^T)(USU^T) = US²U^T

Bottom line: squaring the matrix amounts a squaring of the eigenvalues and the eigenvectors remain the same.

Pulling everything together to a conclusion, the SVD of symmetric matrix A results in the following.

U = V = eigenvectors of A.
S = squares of the eigenvalues of A.

That is, given a symmetric matrix A...

A = USV^T= USU^T = VSV^T
A² = US²V = US²U = VS²V

Is there any practical value in this?

Calculating the SVD of a matrix with a great number of rows or a great number of columns can be pretty expensive in terms of memory consumption (although there are more efficient, abbreviated metods highlighted in the Wikipedia page). On the other hand, the dimensionality of A^TA might be insignificant by comparison.

If the objective is one of, say, calculating eigenvectors of A^TA, is it acceptable to do the SVD of A^TA instead? Some of the literature I've read says a benefit of SVD is that it's more numerically stable than calculating A^TA, but all of the tests I've run in Scilab for cases interesting me seem to work very well. I don't know the answer to this question. Maybe I can find it.

Where's the Shoeshine Boy from Police Squad! when you need him?

I have a final thought that fits with the ideas in this post.

Given a diagonal matrix W and any old matrix A, consider the SVD of W^1/2A.

In this case, V contains the eigenvectors of

(W^1/2A)^T(W^1/2A) =
A^TW^1/2 W^1/2A =
A^TWA

And U contains the eigenvectors of

(W^1/2A)(W^1/2A)^{T =}
W^1/2AA^TW^1/2

If we swap the order and calculate the SVD of AW^1/2

Then V contains the eigenvectors of

(AW^1/2)^T(AW^1/2) =
W^1/2 A^TA W^1/2

And U contains the eigenvectors of

(AW^1/2)(AW^1/2)^T =
(AW^1/2)(W^1/2A^T) =
AW^1/2W^1/2A^T =
AWA^T

This is worthy of further consideration in a future post.

This is post #14 of Linear Algebra for Graphics Geeks. Here are links to posts #1, #2, #3, #4, #5, #6, #7, #8, #9, #10, #11, #12, #13.