BT3017 Interactive Learning Lab

Understanding PCA Through Data, Directions, and Lost Information

Principal Component Analysis, or PCA, starts by looking at the shape of a cloud of points. It finds new directions that follow the main trends in that cloud, starting with the direction of maximum variance. By keeping only the most important of these directions, PCA gives each point a smaller set of new coordinates while preserving most of the important structure. This page lets you see that process step by step using interactive 2D and 3D examples.

See the original data shape Understand why we center data Build intuition for covariance Understand eigenvectors and eigenvalues Choose how many components to keep See what information is lost

PCA in one line

$\Sigma v = \lambda v$

The covariance matrix $\Sigma$ summarizes how the centered variables vary together. Its eigenvectors become the new PCA directions, and its eigenvalues tell us how much of the data cloud's variation each direction captures.

3. Original Data Visualization

What Does the Original Data Look Like?

4. Mean Centering

Why Do We Center the Data?

Before finding the principal directions, we first center the data so the cloud is centered at the origin. PCA should measure the true spread of the cloud, not just where it happens to sit on the graph.

$X_c = X - \mu$

Subtract the mean vector $\mu$ from every data point. This shifts the whole cloud so that it is centered at the origin.

5. Covariance Matrix and Intuition

What Does the Covariance Matrix Mean? ?

The diagonal values show how much each variable spreads on its own. The off-diagonal values show whether centered variables move together, move in opposite directions, or have only a weak linear relationship.

Covariance matrix after centering

6. Principal Components in 2D or 3D

What Do Eigenvectors and Eigenvalues Represent? ? ?

Eigenvectors define the new PCA directions. Their matching eigenvalues tell us how much of the cloud's spread is captured along each direction.

7. Explained Variance and Top x Selection

Why Do We Keep the Top x Components?

PCA keeps the top x components by selecting the eigenvectors with the largest eigenvalues. These directions preserve the biggest patterns in the cloud, so each point can be described with fewer coordinates while keeping most of the important variation.

Scree plot ?

Cumulative explained variance ?

8. Reconstruction and Information Loss

What Information Is Lost When Fewer Components Are Kept? ?

After choosing the top components, each point is projected onto those directions and then mapped back to the original axes. The gap between the original and reconstructed points shows the information that was lost.

Original data cloud

Reconstructed from the selected top components

9. Key Takeaways and Common Misconceptions

What Should You Remember About PCA?

These points summarize the main geometric ideas from the plots and clear up some common misunderstandings about PCA.

Key Takeaways

  • PCA creates new principal directions instead of simply choosing original variables.
  • PC1 follows the direction of maximum variance, and later components capture the remaining spread.
  • Keeping the top x components means selecting the eigenvectors with the largest eigenvalues so each point can be described with fewer coordinates.
  • When components are dropped, the reconstruction becomes less exact, and that gap shows the information loss.

Common Misconceptions

  • PCA is not the same as just deleting columns from the dataset.
  • Each principal component is a linear combination of the original variables.
  • A component with high variance is not automatically the most useful feature for every machine learning task.
  • If all components are kept, PCA changes to new coordinates but does not reduce size or lose information.

Project Notes