Advanced Linear Algebra - Advanced Linear Algebra, Lecture 5.5: Projection and Least Squares

Advanced Linear Algebra, Lecture 5.5: Projection and Least Squares A linear map A is self-adjoint if A*=A. We see two simple but important examples of this: the linear map A*A (for any A), and any orthogonal projection onto a subspace. A projection P is a map satisfying P^2=P, and it is an orthogonal projection if and only if it is self-adjoint. We state and prove an important property of A*A: that it has the same nullspace as A. As a result, if the columns are linearly independent, then A*A is invertible. An application of this is the celebrated "least squares" method from statistics, which in linear algebra terms, amounts to trying to solve an underdetermined system Ax=b (i.e., a "tall and skinny" matrix A). We show that even if Ax=b has no solution, the related system A*Ax=A*b has a unique solution, as long as A has full column rank. Moreover, this solution amounts to finding the orthogonal projection of b onto the range (column space) of A. It follows that for any such A, the projection matrix onto the its columns space is A(A*A)^{-1}A*. Course webpage: http://www.math.clemson.edu/~macaule/math8530-online.html