Hamish Burke | 2025-03-20

Related to: #bigData

PCA

• linear projection of high dimensional data
• reduce dimensions from $p$ to $k$
• orthogonal is dot product between two vectors is 0
  • perpendicular
• covariance between two vars tells us how strongly they are linearly correlated
  • $$\begin{gathered} C_{ij}=\frac{1}{n-1} \\ \sum _{m=1}^n(X_{im-}{\overline{X}}_i)(X_{jm}-{\overline{X}}_j) \end{gathered}$$
  • As we centred it, $\overline{X}$ for both $i$ and $j$ is 0 (simplifies above equation)
• All eigenvectors are perpendicular

1. Centre data (subtract the matrix)

• Subtract the mean of each feature from all the values (to centre it on 0)

2. Calculate $p\ast p$ covariance matrix
3. Calculate the $p$ eigenvectors of the covariance matrix (orthogonal)
4. Select the $k$ eigenvectors that correspond to the highest $k$ eigenvalues to be the new space dimensions