Day 1 - Dimensionality Reduction: PCA, t-SNE, and the Curse of Dimensionality

📌 Key Takeaways

• Curse of dimensionality: as dimensions grow, data becomes sparse and the distances between points become homogeneous, so distance-based algorithms such as KNN and K-means break down.
• PCA is a linear, unsupervised technique that finds orthogonal axes preserving as much variance as possible. Standardization before applying it is mandatory.
• Choose the number of principal components k by cumulative explained variance (e.g., n_components=0.95) or the elbow of the scree plot.
• t-SNE is a nonlinear technique that preserves local structure and is for visualization only. It cannot transform new data, and distances between clusters are meaningless.
• For large datasets use SageMaker's built-in PCA, and when the number of features is very large choose algorithm_mode=randomized.

The Curse of Dimensionality

Week 4 moves into the second pillar of exploratory data analysis (EDA): handling high-dimensional data. As features multiply, a model seems to have more information, but as dimensions grow, the volume of space that the same number of samples must fill grows exponentially. As a result, data points drift apart from one another and distance-based measures lose their discriminating power.

[Low dimension (2D)]             [High dimension (d ≫ 1)]
    ●                              ●    ●    ●
       ● ← close                     ●    ●    ●
              ● ← far              ●    ●    ●
                                   All pairwise distances are similar
  Nearest / farthest distinguishable   d_max / d_min → 1
  → KNN·K-means work normally        → the notion of a neighbor collapses

Vulnerability to High Dimensions, by Algorithm

Algorithms that depend on Euclidean distance, such as KNN and K-means, are especially vulnerable in high dimensions.

The mathematical core of the curse of dimensionality is that, in a high-dimensional unit hypercube, the ratio of the maximum to the minimum distance between two random points converges to 1. In other words, the "nearest neighbor" and the "farthest neighbor" become effectively the same. For this reason, dimensionality reduction does not merely cut computation; it also restores the space in which distance-based learning works.

PCA: Principal Component Analysis

PCA is a linear dimensionality-reduction technique that finds orthogonal axes (principal components) that preserve as much variance as possible and projects the data onto those axes.

• The first principal component (PC1) is the direction of greatest variance
• The second principal component (PC2) is orthogonal to PC1 and has the greatest remaining variance
• Keep only the top k principal components to reduce dimensions from d → k

Key precondition: always standardize before PCA. Because it is variance-based, variables with large scales dominate the principal components.

Preprocessing Requirements Before PCA

import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
 
X_scaled = StandardScaler().fit_transform(X_train)
 
pca = PCA(n_components=0.95)   # minimum number of components explaining 95% of the variance
X_reduced = pca.fit_transform(X_scaled)
 
print("Selected number of components:", pca.n_components_)
print("Explained variance ratio:", pca.explained_variance_ratio_.cumsum())

If you pass a 0–1 ratio to n_components instead of an integer, it automatically selects "the minimum number of principal components whose cumulative explained variance exceeds that ratio."

How to Choose the Number of Components k

Choose an appropriate k from the scree plot or the elbow of the cumulative explained variance curve.

PCA is equivalent to an eigendecomposition of the data's covariance matrix. The eigenvectors are the principal component directions, and the eigenvalues are the variance in those directions. So the "explained variance ratio" is simply each eigenvalue divided by the sum of all eigenvalues. Singular value decomposition (SVD) yields the same result, and scikit-learn and SageMaker use randomized SVD on large data to improve efficiency.

⚠️ Pitfall: PCA is an unsupervised transform that never looks at the target. "The direction of large variance" is not always "the direction useful for prediction." If a decisive signal hides in a low-variance axis, PCA may throw it away.

⚠️ Pitfall: Principal components are linear combinations of the original features, so physical meaning is lost. If a question says "you must explain which variables were used for prediction," the answer is feature selection, not PCA.

SageMaker Built-in PCA Algorithm

Large datasets cannot be loaded into memory entirely, so use SageMaker's built-in PCA algorithm.

• Input is recordIO-protobuf or CSV, and training supports distributed processing
• Use the resulting principal components to reduce the input dimension of downstream models (XGBoost, etc.)

from sagemaker import image_uris
 
container = image_uris.retrieve("pca", region="ap-northeast-2")
# Example Estimator hyperparameters
hyperparameters = {
    "feature_dim": 784,
    "num_components": 50,
    "algorithm_mode": "randomized",
    "subtract_mean": "true",
}

AWS Service Mapping

t-SNE: A Nonlinear Technique for Visualization Only

t-SNE (t-distributed Stochastic Neighbor Embedding) is a nonlinear technique that maps data so that neighbor relationships (local structure) in high dimensions are preserved in 2–3 dimensions.

t-SNE is excellent for seeing clusters with the eye, but it is not used as preprocessing for model input. Distances between clusters and cluster sizes have no physical meaning, and new samples cannot be projected into the same space.

from sklearn.manifold import TSNE
 
tsne = TSNE(n_components=2, perplexity=30, random_state=42)
X_embedded = tsne.fit_transform(X_scaled)   # 2D coordinates for visualization

perplexity balances how many neighbors to consider and is typically tuned in the range 5–50.

t-SNE converts high-dimensional distances into conditional probability distributions and minimizes the KL divergence so that a heavy-tailed t-distribution in low dimensions reproduces the same relationships. The heavy tails of the t-distribution relieve the "crowding problem," letting clusters spread out instead of clumping. Because of this process, however, the size of the empty space between clusters is meaningless — on the exam, an option that says "judge similarity by the distance between t-SNE clusters" is a trap.

t-SNE has only fit_transform; it has no transform that stores the learned mapping and applies it to new samples. An option that says "put new data arriving in production into the same embedding space" is wrong.

Dimensionality Reduction Technique Selection Table

Reduce or Select?

You need to reduce dimensions
├─ Must the result be explained variable by variable? (regulated industry, audit response)
│    ├─ Yes → Feature selection (keep the original columns as they are)
│    └─ No ↓
├─ Is the goal "check by eye whether clusters exist"?
│    ├─ Yes → t-SNE (2–3D, visualization only. Never as model input)
│    └─ No ↓
├─ Are relationships mostly linear?
│    ├─ Yes → PCA (standardize → choose k by cumulative explained variance)
│    └─ No  → Nonlinear compression such as an autoencoder
└─ Does the data exceed single-instance memory?
     └─ Yes → SageMaker built-in PCA (randomized if there are many features)

Summary of Preprocessing Techniques for High Dimensions

Dimensionality reduction is only one solution to the high-dimensional problem. The exam asks not "reduce the dimensions" but "what fits this situation."

[High-dimensional raw data]
        │
   ① Split (train / test)  ← from here on, every fit uses the training set only
        │
   ② Missing-value imputation · encoding
        │
   ③ Standardization (StandardScaler)
        │
   ④ ┌─ Interpretation needed ─→ Feature selection (keep original columns)
     └─ Not needed ─→ PCA (n_components=0.95)
        │
   ⑤ Model training (XGBoost / linear / neural network …)
        │
   ⑥ On the test set, apply only transform with what was learned in ②–④

Question Clue → Answer Mapping

The next post covers feature selection, which reduces dimensions while keeping the original features, and multicollinearity.

📖 Glossary

• Curse of dimensionality: The phenomenon in which, as dimensions grow, data becomes sparse and distance discrimination disappears, making learning harder.
• Sparsity: Data becoming thin because the same number of samples must fill a much larger space.
• Principal component: A new coordinate axis taken, in order, along the directions of greatest variance in the data. They are mutually orthogonal.
• Projection: Moving the original data onto new axes, "casting a shadow," and re-assigning coordinates.
• Explained variance ratio: The share of the total variance that each principal component holds.
• Scree plot: A graph with the principal component number on the x-axis and explained variance on the y-axis. Pick k at the bend (elbow).
• Eigendecomposition / SVD: Linear-algebra operations that extract principal component directions (eigenvectors) and their variances (eigenvalues) from the covariance matrix. SVD is a way to get the same result more stably.
• Local structure / global structure: Relationships among nearby neighbors / the overall large-scale layout of the data. t-SNE preserves the former well, PCA the latter.
• perplexity: A value in t-SNE that controls "how many neighbors to look at." Usually set between 5 and 50.
• recordIO-protobuf: A binary input format preferred by SageMaker built-in algorithms. More efficient than CSV for large volumes.