Day 2 - SageMaker Builtin 1: XGBoost, Linear Learner, K-Means, KNN

📌 Key Takeaways

• "Tabular data + high accuracy" is almost always XGBoost. It is tree-based, so it needs no scaling and handles missing values on its own.
• "Massive scale + fast + linear/interpretable" is Linear Learner. It trains several models at once and automatically picks the best one.
• "Discover groups/segments without labels" is K-Means. The number of clusters k must be specified in advance.
• Distance-based and linear families (KNN, K-Means, Linear Learner) require scaling, while trees (XGBoost) do not. An option that says "skip scaling" is a wrong-answer signal for distance/linear families.
• For an XGBoost overfitting scenario, the correct direction is max_depth↓, eta↓, regularization (gamma/lambda/alpha)↑, subsample↓.

The Four Algorithms at a Glance

Building on yesterday's mapping table, today we dig deep into the four most common tabular data algorithms. These are core builtins within the exam scope, and the key is to distinguish which problem each one fits, its input formats, and its key hyperparameters.

                  [Tabular data problem]
                          │
              Are there labels?
        ┌─────────Yes────────┴────────No────────────┐
        │                                          │
   Output is category/number                  Discover groups
        │                                          │
 ┌──────┴───────┐                              K-Means
 │              │                            (specify k up front,
Is the          "Decide by the                 scaling required)
relationship    most similar
nonlinear?      cases"
 │                    │
 ├─ Yes → XGBoost     └→ KNN
 │   (no scaling needed)   (scaling required,
 └─ No/massive/interpretable   weak in high dimensions)
      → Linear Learner
        (scaling needed)

XGBoost — The Workhorse for Tabular Data

A gradient boosting tree algorithm. It is strong for both classification and regression, and in practice, on exams, and on Kaggle it is effectively the default for tabular data.

• Use: Binary/multiclass classification, regression, ranking
• Input formats: CSV, libsvm, Parquet, RecordIO-protobuf (for CSV, the first column is the label)
• Strengths: Handles missing values on its own, automatically captures nonlinearity and interactions, and needs no scaling because it is tree-based

Key hyperparameters:

objective         binary:logistic | multi:softmax | reg:squarederror
num_round         number of boosting rounds (number of trees)
max_depth         tree depth — larger means more complex, higher overfitting risk
eta               learning rate (smaller converges more slowly, typically 0.01~0.3)
subsample         row sampling ratio (suppresses overfitting)
colsample_bytree  column sampling ratio
gamma, lambda, alpha   regularization (suppress overfitting)
scale_pos_weight  class imbalance correction

XGBoost overfitting control is a combination of knobs that "simplify the model." Reduce max_depth; lower eta while raising num_round; add randomness with subsample/colsample_bytree; and apply regularization with gamma/lambda/alpha. On the exam, a scenario like "training loss keeps falling but validation loss rises (overfitting)" usually has the answer in the direction of max_depth↓ / eta↓ / regularization↑ / subsample↓. Underfitting is the opposite direction.

XGBoost Tuning Direction Table

⚠️ Trap: If you only lower eta and leave num_round unchanged, training ends before the model has learned enough, causing underfitting. eta↓ and num_round↑ move as a pair.

Linear Learner — Linear Model, Massive Scale, Fast

SageMaker's distributed and optimized version of linear/logistic regression. It supports both classification and regression.

• Use: Binary/multiclass classification, regression
• Input formats: RecordIO-protobuf (recommended, efficient), CSV
• Strengths: Fast on huge data, and it trains multiple models at once and automatically selects the best one. Easy to interpret
• Caution: Because it is a linear model, feature scaling is decisive (the builtin provides a normalization option)

Key hyperparameters:

predictor_type        binary_classifier | multiclass_classifier | regressor
num_classes           number of classes for multiclass classification
mini_batch_size       mini-batch size
learning_rate         learning rate
l1, wd(L2)            regularization
normalize_data        whether to standardize the input
balance_multiclass_weights   imbalance correction

The fork between XGBoost and Linear Learner: if nonlinear relationships matter, choose XGBoost; if the relationship is mostly linear, the scale is huge, and interpretability matters, choose Linear Learner.

K-Means — Unsupervised Clustering

It splits data into k groups without labels. It is used for things like customer segmentation.

• Use: Clustering (unsupervised)
• Input formats: RecordIO-protobuf, CSV
• Key point: The number of clusters k must be specified in advance
• Caution: It is distance-based, so scaling is mandatory

Key hyperparameters:

k                 number of clusters (required)
feature_dim       feature dimension
mini_batch_size   mini-batch size
init_method       random | kmeans++ (initial center selection)
extra_center_factor   multiplier for generating extra candidate centers and then reducing them

There is no single correct answer for choosing k, so the following methods are used.

SageMaker K-Means uses extra_center_factor to generate generous candidate centers and then reduce them to find stable clusters.

KNN — Distance-Based Classification/Regression

It predicts by majority vote (classification) or average (regression) of the k nearest neighbors.

• Use: Classification, regression
• Input formats: RecordIO-protobuf, CSV
• Key point: It is closer to "memorization" than to learning. Because neighbor search is costly at inference time, SageMaker accelerates it with dimensionality reduction (sample_size, dimension_reduction)
• Caution: Because it is distance-based, scaling is mandatory. Performance drops sharply in high dimensions (the curse of dimensionality)

Key hyperparameters:

k                 number of neighbors to consult
sample_size       number of samples used for training
dimension_reduction_type   sign | fjlt (dimensionality reduction method)
predictor_type    classifier | regressor

The curse of dimensionality is the phenomenon where, as dimensions increase, data scatters across the space and the distance gap between the "nearest neighbor" and the "farthest neighbor" shrinks. Distance itself loses discriminating power, so the performance of distance-based algorithms such as KNN collapses. This is why SageMaker KNN offers dimension_reduction_type (sign, fjlt), and reducing dimensions beforehand with PCA (Day 4) serves the same purpose.

Distance-based algorithms (KNN, K-Means) and linear models (Linear Learner) are directly distorted by feature scale, so scaling is mandatory. For example, if you feed "salary (in millions)" and "age (in tens)" without scaling, Euclidean distance is dominated by salary. Tree-based models (XGBoost), on the other hand, split each feature independently on a threshold, so they are invariant to scale. When the exam asks "is scaling needed?", answer immediately: needed for distance/linear families, not needed for tree families.

Preprocessing Requirements Summary

Evaluation Metrics for the Four Algorithms

Even for the same algorithm, the metric to look at differs depending on predictor_type/objective.

⚠️ Trap: An option that says to evaluate clustering results with "accuracy" is wrong. K-Means is unsupervised, so there are no ground-truth labels to compare against. Evaluate only with internal metrics (silhouette, WCSS) or domain interpretation.

Handling Class Imbalance

Common Operational Tips

• RecordIO-protobuf is the most efficient input format for most builtins (streaming training is possible in Pipe mode).
• For large data, streaming from S3 in Pipe mode lets training start faster without copying to disk.
• Hyperparameters can be explored with SageMaker Automatic Model Tuning (a Week 7 topic).

[S3 training data]              [Builtin container]          [Model artifact]
 CSV / libsvm /      ──File─▶   Specify hyperparameters ──▶  s3://.../model.tar.gz
 Parquet /            or        (objective, k, eta …)               │
 RecordIO-protobuf   ──Pipe─▶   Early stopping on a                 ▼
      │                         validation channel          Real-time endpoint
 (Do scaling and encoding                                    or batch transform
  beforehand in a Processing Job)

• Finish scaling and encoding in a front-end preprocessing step, not in the training job, and reuse the same logic at inference time so that no training-serving mismatch arises.

Question Clue → Answer Mapping Table

K-Means and KNN have similar-looking names and often get mixed up in the options. K-Means is unsupervised clustering (no labels); KNN is supervised prediction (labels required). If you choose based on the letter "k" alone, you will be wrong.

Tomorrow we cover the specialized builtins used when the data has a special form, such as text, images, time series, and recommendations.

📖 Glossary

• Gradient boosting: An ensemble learning method that stacks weak trees sequentially, with each new tree correcting the errors of the previous ones.
• libsvm format: A sparse-data text format of the form label index:value index:value that records only the non-zero values.
• RecordIO-protobuf: The binary input format preferred by SageMaker builtins. Files are small and well suited to Pipe mode streaming.
• Pipe mode: An input method that streams training data directly from S3 instead of copying it to the instance disk.
• eta (learning rate): The ratio that decides how much each boosting round updates the model. The smaller it is, the slower and more stably it converges.
• subsample / colsample_bytree: The fraction of rows and columns used when building each tree. Adds randomness to reduce overfitting.
• scale_pos_weight: An XGBoost parameter that raises the weight of the positive class to correct class imbalance.
• kmeans++: An initialization method that picks initial centers far apart from each other. More stable than random initialization.
• Elbow (WCSS): A method of choosing k at the point where the drop in within-cluster sum of squares flattens as k increases.
• Curse of dimensionality: The phenomenon where distances between points become similar as dimensions grow, so distance-based algorithms lose discriminating power.