What do the eigenvalues/eigenvectors of a matrix mean?

Your question would probably be better asked at mathoverflow.net, but that forum can be a little scary to wade into. I'd suggest posting there anyway, but I'll also give a description, albeit not rigorously.

Analyzing eigenvectors & eigenvalues of a matrix in the simple case of a matrix-vector product can give a lot of information about how that matrix will skew a vector. In a sense, you can imagine each eigenvector grabbing hold of a rubber sheet -- the vector 'x' in the product Ax -- and pulling it in some direction. Each eigenvector pulls in that direction, and the eigenvalue controls the strength of that pull.

The magnitude of the eigenvector itself can potentially be anything, but it's fairly straightforward to prove that a matrix factorization PDP^-1 with scaled eigenvectors is equivalent to another matrix factorization QEQ^-1 with unit eigenvectors; the difference being their eigenvalues (note: PDP^-1 = QEQ^-1 = A).

This is at the heart of proving that an eigen decomposition of a matrix (into the form PDP^-1 or QLQ^-1) is a non-unique factorization of the matrix A. Hence, it's not at all uncommon to see the eigenvectors be expressed as unit vectors.

In the 'best' case from the standpoint of numerical linear algebra, you'd want to have a full rank matrix A with orthogonal (perpendicular) or orthonormal (perpendicular unit vectors) eigenvectors.

However, in reality it can be common to have a matrix without full rank (which corresponds to one or more eigenvalues being zero), or matrices with nearly parallel eigenvectors which makes the matrix nearly singular or nearly non-invertible (which corresponds to the matrix having a high condition number).

Matrices with a high condition number are more unstable, and when used with iterative methods can have a higher tendency to exhibit pathological behavior in whatever algorithm their used in. Behavior in the direction of the other non-conflicting eigenvectors will be less unstable, but in the directions of the eigenvectors that are nearly parallel things will be far more erratic.

As an example, there are a number of iterative methods for attempting to solve Ax=b without using least squares or directly inverting A. When 'x' points in a direction that is also nearly parallel to the eigenvectors that are nearly parallel to eachother, the algorithm will take more time to converge than if it pointed in a more friendly direction -- if it converges at all.

-Brian

Complementing Brian's answer, some eigenvalues/eigenvectors of special matrices have special properties, and knowing some of these can save you a lot of trouble understanding papers.

One that is very common in machine learning is the equilibrium distribution of a Markov chain. A Markov chain over a discrete state space is a probability distribution on sequences of states such that the next state depends only on the current state. Given an initial state distribution, a Markov chain is specified fully by a transition matrix, that is, a matrix M such that M[i,j] = P(j|i). Trivially, if you multiply M by the column vector that is 1 only on the current state you have the distribution of states that follow that first state. Some Markov chains are reversible and stationary (there is always a positive probability to from each state to each other state through some path in the matrix). These Markov chains have the property that they have an equilibrium distribution, that is, after running for some time, regardless of the initial state, the probability of finding the chain in a certain state is always the same. Hence, if p is a vector such that p[i] the stationary probability of M being in state i, then Mp = p; that is, p is an eigenvector of M with eigenvalue 1. This leads to all sorts of interesting algorithms, such as PageRank.

Another very common eigenvalue/eigenvector in machine learning is the one (or ones) used in spectral clustering. There there is no clear intuition as to why it works, but if you build a specific matrix from a graph (called the graph laplacian matrix), the eigenvectors with smallest eigenvalues (except for the eigenvector with the smallest eigenvalue) can be used to cluster nodes in the graph in a way that approximates the best normalized cut.

Also, eigenvalues and eigenvectors can be related to matrix approximation. Since, as Brian said in his answer, the eigenvalues represent how much the matrix "stretches" vectors in the direction of the eigenvectors, if the matrix has some big eigenvalues and lots of very small eigenvalues then you can essentially replace it with V S V^T, where V is a rectangular matrix with one eigenvector per column, S is a small diagonal matrix with the corresponding eigenvalues and V^T is V transposed. If the matrix has some very big and lots of very small eigenvalues, you can use just the eigenvectors with large eigenvalues in V and obtain essentially a much smaller representation of the matrix. This is the idea behind PCA, LSI, and many other forms of dimensionality reduction (specially because this truncated approximation is the best possible low-rank approximation of the original matrix).

Last, eigenvalues and eigenvectors can be used to compute some important functions of the matrix. The determinant is the product of the eigenvalues, and the trace is the sum of the eigenvalues.

Try this out...you will feel what the concept means.

http://www.physlink.com/education/askexperts/ae520.cfm