Day 4 - Unsupervised Learning and Anomaly Detection: RCF, PCA, IP Insights, Topic Models (LDA/NTM)

📌 Key Takeaways

• All four builtins today work with structure, anomalies, dimensions, and topics without labels. The signals in a question are "there are no labels / rare events / too many dimensions / discover topics."
• RCF is for anomaly detection on general numeric data, and IP Insights is dedicated to "entity-IP" relationship anomalies. Do not mix the two.
• For real-time stream anomaly detection, think of the RANDOM_CUT_FOREST function in Kinesis Data Analytics.
• PCA compresses many correlated features while preserving variance. For very large dimensions, use algorithm_mode=randomized.
• LDA in SageMaker is a topic model (Latent Dirichlet Allocation) and shares only a name with Linear Discriminant Analysis, the dimensionality reduction technique. Choose NTM for large scale/GPU, and LDA for simplicity/interpretability.

Unsupervised Builtins at a Glance

              [There are no labels]
                     │
   ┌──────────┬──────┴───────┬──────────────┐
"Find rare   "Too many       "Find topics   "Group similar
 events"      features"       in documents"   things together"
   │             │              │              │
What data?     PCA          Document scale?   K-Means
   │        (regular /          │            (Day 2)
   ├─ General numeric randomized) ├─ Large/GPU → NTM
   │  → RCF                   └─ Small/interpretable → LDA
   │  (if real-time, Kinesis
   │   Data Analytics RCF)
   └─ (entity, IP) pairs
      → IP Insights

Random Cut Forest (RCF) — Anomaly Detection

It builds trees by cutting the data at random and assigns a high anomaly score to points that are easily isolated (rare).

• Use: Anomaly detection (unsupervised). Time series spikes, rare patterns
• Input: CSV, RecordIO-protobuf
• Output: Per-record anomaly score (higher means more anomalous)
• Streaming: Kinesis Data Analytics has a built-in RANDOM_CUT_FOREST function, which enables real-time anomaly detection

Key hyperparameters:
  num_trees              number of trees (more means more stable)
  num_samples_per_tree   samples per tree
  feature_dim            feature dimension

The intuition behind RCF is that "anomalies get isolated faster than normal points." Normal points sit in dense regions and need many cuts to be separated, but a rare point is isolated with just a few random cuts. This isolation depth becomes the score. It needs no labels and reacts even to new kinds of anomalies, so it represents the "there are almost no labels / the anomalies keep changing" problem mentioned on Day 1. If the requirement is real-time, think of the RCF SQL function in Kinesis Data Analytics.

[Normal point]  Dense region → many cuts needed to separate → deep isolation depth → low score
   ● ●●●
   ●●●●●   ──┬── ──┬── ──┬── ──┬──  (needs 4 cuts to separate)
   ●●●

[Anomalous point]  Sparse region → isolated in a few cuts → shallow isolation depth → high score
                ○     ──┬──               (separated in 1 cut)

Anomaly Detection: Supervised vs Unsupervised Decision Table

⚠️ Trap: For an anomaly detection problem, choosing "train it as a normal/anomaly binary classification" is correct only if the question gives grounds that enough anomaly labels exist. Choosing supervised classification without such grounds is wrong.

PCA — Dimensionality Reduction

It compresses many highly correlated features into a small number of principal components that preserve as much variance as possible.

• Use: Dimensionality reduction (unsupervised). Visualization, lightening the input to downstream models, mitigating multicollinearity
• Input: RecordIO-protobuf, CSV
• Modes: regular (medium dimensions), randomized (very large dimensions, approximate, fast)

Key hyperparameters:
  num_components     number of principal components to keep
  algorithm_mode     regular | randomized
  subtract_mean      remove the mean (centering)

• It is scale-sensitive, so you usually scale and center first.
• PCA is unsupervised and does not use labels. If you need a dimensionality reduction that uses labels, that is LDA (Linear Discriminant Analysis), but there is a catch — LDA in SageMaker is a topic model (Latent Dirichlet Allocation) and shares only the name.

⚠️ Trap: If you skip scaling before PCA, a feature with large units (e.g., salary) monopolizes the variance, and that feature's direction becomes the first principal component. Standardizing before PCA is almost always the correct option.

IP Insights — Entity-IP Anomaly Detection

It learns the association between an entity (user/account) and IPv4 addresses and detects combinations that differ from the usual.

• Use: Abnormal logins, account takeover, bot detection (unsupervised)
• Input: CSV — (entity, IP) pairs
• Output: Likelihood that the entity would use that IP (lower means more anomalous)
• Technique: Learns entity-IP patterns with neural network embeddings and contrasts them against random negative samples

Key hyperparameters:
  num_entity_vectors   size of the entity embedding hash space
  vector_dim           embedding dimension
  num_ip_encoder_layers
  random_negative_sampling_rate

The fork between RCF and IP Insights: both are anomaly detection, but RCF is for anomalies in general numeric data, while IP Insights specializes in the "entity-IP" relationship. For a login from an unusual IP, use IP Insights; for a sudden change in a general metric, use RCF.

Real-Time vs Batch Anomaly Detection Paths

If you see only the word "real-time" and always choose a SageMaker real-time endpoint, you will be wrong on a question that says to process within the streaming pipeline. If there is a clue that the data arrives through a Kinesis stream, Kinesis Data Analytics RCF is the more direct answer.

Topic Models: LDA and NTM

They find latent topics and per-topic word distributions in documents without labels. Both are topic models, but their implementations differ.

• Common input: Bag-of-Words (frequency counts per word index), RecordIO-protobuf or CSV
• The user specifies the number of topics (LDA: num_topics, NTM: num_topics)

Topic models assume that "each document is a mixture of several topics, and each topic is a distribution over words." LDA estimates this with a Bayesian probabilistic model, and NTM approximates the same goal with a neural network. Both are unsupervised, so there are no ground-truth topic labels, and the number of topics is a hyperparameter. When the exam asks "which of the two?", decide by data scale, GPU, and interpretability (large scale/GPU = NTM, simple/interpretable = LDA). This LDA (the topic model) shares only the name with Linear Discriminant Analysis and is completely different.

An option saying "LDA is a dimensionality reduction algorithm" is wrong in the SageMaker context. It is a classic trap that targets the abbreviation collision, and the SageMaker builtin LDA is a topic model.

Preprocessing Requirements and Evaluation Methods

Unsupervised algorithms have no ground-truth labels, so they cannot be evaluated by "accuracy." What you judge them by is the exam point.

An option that says to evaluate unsupervised results by "accuracy" is wrong, because there are no ground-truth labels. Judge RCF by the score threshold, PCA by explained variance, and topic models by perplexity and interpretability.

Pipeline That Puts PCA in Front of Other Algorithms

[Original 500 features]
      │  Standardize (mean 0, variance 1)
      ▼
[PCA  algorithm_mode=randomized, num_components=30]
      │  30 principal components (mutually orthogonal, 95% variance preserved)
      ▼
[Downstream]  KNN (eases curse of dimensionality) / K-Means (stabilizes distances) / Visualization (2~3 components)

• With fewer dimensions, training and inference of distance-based models become faster and more stable.
• The cost is loss of interpretability. If a regulatory review requires explaining "which features had an influence," PCA is at a disadvantage.

Question Clue → Answer Mapping Table

Tomorrow we synthesize all of Week 6 from an algorithm-selection decision-making perspective.

📖 Glossary

• Anomaly score: A numeric value showing how different a record is from the usual. For RCF, a larger value means more anomalous.
• Isolation: Leaving only a single point through random cuts. The more anomalous a point is, the fewer cuts it takes to isolate it.
• Principal component: A new axis in the direction that explains the most variance in the data. They are mutually orthogonal.
• Multicollinearity: A phenomenon where features are strongly correlated with each other, making the coefficient interpretation of linear models unstable. PCA mitigates it.
• subtract_mean (centering): A preprocessing step that subtracts the mean from each feature to align the origin. It is a prerequisite for PCA.
• Entity: In IP Insights, the subject observed paired with an IP, such as a user ID or account ID.
• Random negative sampling: A technique that artificially creates (entity, IP) combinations that were never actually observed and uses them for contrastive learning.
• Bag-of-Words (BoW): A representation that discards word order and counts only occurrence frequencies to turn a document into a vector. It is the input to topic models.
• num_topics: The number of topics a topic model will find. With no ground truth, it is a hyperparameter the user sets.
• LDA name collision: The SageMaker builtin LDA is a topic model (Latent Dirichlet Allocation), and shares only the abbreviation with Linear Discriminant Analysis, the dimensionality reduction technique.