Motivation I Data Compression
- compress the data, use up less computer memory or disk space
- speed up our learning algorithms.
Motivation II Visualization
If you have 50 features, it’s very difficult to plot 50-dimensional data. But if you reduce dimensions, the problems is what these new features means.
Principal Component Analysis Problem Formulation
By far the most commonly used algorithm is something called principal components analysis or PCA. What PCA does is it tries to find the surface onto which to project the data so as to minimize that. Before applying PCA it’s standard practice to first perform mean normalization and feature scaling.
Principal Component Analysis Algorithm
Reduce the dimensions :
- Mean normalization, maybe perform feature scaling as well
- Covariance matrix, [latex]\sum = \frac {1}{m} \sum _{i=1}^{n}(x^{(i)})(x^{(i)})^T[/latex]
- Eigenvectors of the matrix sigma
Choosing The Number Of Principal Components
The variation of the training sets : [latex]\frac {1}{m} \sum_{i=1}^{m} \left \ x^{(i)} \right \^2[/latex] Try to choose k, a pretty common rule of thumb for choosing k is to choose the smaller values so that the ratio of the average square projection error and the total variation in the data between these is less than 0.01. [latex]\frac {\frac {1}{m} \sum_{i=1}^{m} \left \ x^{(i)} - x^{(i)}_{approx} \right \^2}{\frac {1}{m} \sum_{i=1}^{m} \left \ x^{(i)} \right \ ^2} = 1 - \frac {\sum_{i=1}^{k}S_{ii}}{\sum_{i=1}^{n}S_{ii}} \leq 1 \%[/latex] [latex]\frac {\sum_{i=1}^{k}S_{ii}}{\sum_{i=1}^{n}S_{ii}} \geq 99 \%[/latex] After compressed : [latex]x^{(i)}_{approx} = U_{reduce}Z^{(i)}[/latex]
Advice for Applying PCA
- Don’t think of PCA as a way to prevent over-fitting. A much better way to address it, to use regularization. And the reason is that it throws away or reduces the dimension of your data without knowing what the values of y is so that it might throw away some valueable information.
- First consider doing it with your original raw data [latex]x^{(i)}[/latex], and only if that doesn’t do what you want, then implement PCA before using [latex]Z^{(i)}[/latex].