Day 2 - Hyperparameter Tuning (AMT): Bayesian, Random, Hyperband

📌 Key Takeaways

• Parameters are values that are learned; hyperparameters are values a person sets before training. AMT searches the space of the latter to find the combination with the best validation metric.
• Strategy choice: Bayesian if you want efficiency with few evaluations, Hyperband for pruning epoch-iterative deep learning, Random for a simple, fully parallel baseline, and Grid for exhaustively covering a few categorical combinations.
• The objective metric is the optimization direction. On imbalanced data, maximizing Accuracy gives 99% even for "all negative," making the model meaningless — use F1, AUC, or Recall.
• A custom script has no standard metric names, so AMT recognizes the metric only if you extract it from the logs with a metric_definitions regex.
• Raising max_parallel_jobs too much reduces the benefit of Bayesian search (because candidates are chosen without seeing earlier results). Random is independent, so it loses nothing.

Hyperparameters vs Parameters

If yesterday you learned how to run one training job well, today covers SageMaker Automatic Model Tuning (AMT, the Hyperparameter Tuning Job), which runs that training many times automatically to search for the best combination.

Define the search space → run several training jobs (trials) → evaluate each trial's objective metric
→ pick the next trial's hyperparameters according to the strategy → return the best combination

Hyperparameter tuning is essentially "black-box optimization." The objective function (the validation metric) is not differentiable, and one evaluation (= one training run) is expensive. So strategies that find a good region with as few evaluations as possible (Bayesian, Hyperband) are more efficient than random or grid search.

Execution Structure of a Tuning Job

[HyperparameterTuner]
   │  Search space: eta ∈ [0.01, 0.3], max_depth ∈ [3, 10]
   │  Objective metric: validation:auc (Maximize)
   ▼
 ┌──────────── runs max_parallel_jobs at the same time ────────────┐
 │  trial#1 ──train──→ auc 0.81                                    │
 │  trial#2 ──train──→ auc 0.86  ← promising                       │
 │  trial#3 ──train──→ auc 0.74  ← hopeless (early-stop candidate) │
 └─────────────────────────────────────────────────────────────────┘
   │  The strategy (Bayesian, etc.) looks at the results and picks the next candidates
   ▼  … repeat until max_jobs is used up …
[Return the best combination + BestTrainingJob]

Four Search Strategies

The strategies AMT supports and their characteristics.

from sagemaker.tuner import HyperparameterTuner, ContinuousParameter, IntegerParameter
 
tuner = HyperparameterTuner(
    estimator=est,
    objective_metric_name="validation:auc",
    objective_type="Maximize",
    hyperparameter_ranges={
        "eta": ContinuousParameter(0.01, 0.3),
        "max_depth": IntegerParameter(3, 10),
    },
    strategy="Bayesian",      # Bayesian | Random | Grid | Hyperband
    max_jobs=30,
    max_parallel_jobs=3,
)
tuner.fit({"train": train_s3, "validation": val_s3})

Discriminating signals:

• "Efficient search with few training runs" → Bayesian.
• "Like deep learning, iterates over epochs + we want to discard hopeless candidates quickly" → Hyperband.
• "Simple/fully parallel, baseline" → Random.
• "Exhaustive over a few categorical combinations" → Grid.

⚠️ Trap: Grid search looks safe because it "looks at everything," but the number of combinations explodes as soon as even a few continuous hyperparameters are mixed in. If the question features a continuous range (ContinuousParameter), Grid is effectively a wrong-answer candidate.

Objective Metric

AMT extracts the metric from each trial's logs with a regex and compares them.

• Built-in algorithms provide standard metric names as-is (validation:auc, validation:rmse, etc.).
• For a custom script, you must specify with metric_definitions a regex that parses the value from the logs.

metric_definitions=[
    {"Name": "validation:f1", "Regex": "val_f1: ([0-9\\.]+)"}
]

• objective_type: Maximize (AUC, F1, Accuracy) or Minimize (RMSE, Loss).
• Since the metric choice is "what you optimize," on imbalanced data it is right to target F1/AUC rather than Accuracy.

Objective Metric Interpretation Table by Situation

Tuning optimizes only in the direction the objective metric points. On data with 1% positives, such as fraud detection, maximizing Accuracy gives 99% even if you label everything "normal," making the model meaningless. That is why choosing the objective metric to fit the business goal and the data imbalance (F1, Recall, AUC, etc.) is the precondition for tuning to succeed.

⚠️ Trap: If you use a custom container or script and omit metric_definitions, AMT cannot read the metric. In that case, options that raise instance_count or change the input mode are all irrelevant wrong answers.

Early Stopping

It saves cost by stopping hopeless trials early. Distinguish two levels.

tuner = HyperparameterTuner(
    ..., early_stopping_type="Auto",   # Off | Auto
)

Discriminating signal: "automatically stop hopeless training to cut tuning cost" → early stopping (Auto) or Hyperband.

Warm Start

A new tuning job starts by leveraging the results of a previous tuning job. Instead of searching from scratch, it starts from the good regions already discovered.

from sagemaker.tuner import WarmStartConfig, WarmStartTypes
 
warm = WarmStartConfig(
    warm_start_type=WarmStartTypes.IDENTICAL_DATA_AND_ALGORITHM,
    parents={"previous-tuning-job-name"},
)
tuner = HyperparameterTuner(..., warm_start_config=warm)

Discriminating signals:

• "Continue the last tuning and explore further / data and algorithm unchanged" → IDENTICAL_DATA_AND_ALGORITHM.
• "The data changed slightly but we want to reuse the knowledge from the last tuning" → TRANSFER_LEARNING.

In a question that says "continue the previous search," options such as raising max_parallel_jobs, switching the strategy to Grid, or turning off early stopping get mixed in. All three are unrelated to reusing the previous job's search knowledge — warm start is the only means of reuse.

The Trade-off Between Parallelism and Bayesian Efficiency

• A large max_parallel_jobs finishes sooner, but Bayesian search decides the next candidates by looking at earlier results, so making parallelism too large reduces learning efficiency (use of information).
• Random loses no efficiency when you raise parallelism (the trials are independent of each other).
• On the exam, "it was Bayesian but maximizing parallelism did not improve things as expected" is answered by this trade-off.

max_parallel_jobs = 1        max_parallel_jobs = N (large)
  trial1 → result applied      trial1 ┐
  trial2 → result applied      trial2 ├─ decided at the same time
  trial3 → result applied      trial3 ┘  (cannot see earlier results)
  ─────────────────────        ─────────────────────
  Slow / best efficiency per evaluation   Fast / Bayesian benefit reduced
  Random has the same efficiency either way (each trial is independent)

Tuning Strategy Decision Tree

We need to find hyperparameters automatically
├─ Does training iterate over epochs/rounds and expose intermediate performance?
│    ├─ Yes + cost savings matter → Hyperband (early pruning built in)
│    └─ No ↓
├─ Is one training run expensive?
│    ├─ Yes → Bayesian (+ early_stopping_type="Auto", keep parallelism modest)
│    └─ No ↓
├─ Is the search space a few categorical combinations?
│    ├─ Yes → Grid (exhaustive search)
│    └─ No  → Random (simple baseline, fully parallel)
└─ Is there a previous tuning result?
     ├─ Same data and algorithm → warm start IDENTICAL_DATA_AND_ALGORITHM
     └─ Slightly changed        → warm start TRANSFER_LEARNING

Question Clue → Answer Mapping

The next post covers overfitting and underfitting, which block the goal of tuning — "good generalization" — and how to respond (regularization and data augmentation).

📖 Glossary

• Hyperparameter: a setting a person decides before training starts. Learning rate, tree depth, regularization strength, and so on.
• Parameter: a value determined automatically from the data during training. Neural network weights, regression coefficients, and so on.
• AMT (Automatic Model Tuning): a SageMaker feature that runs training many times with different hyperparameter combinations to find the best one.
• Trial (one training job): an individual training job a tuning job runs. The unit that evaluates one hyperparameter combination once.
• Objective metric: the reference metric used to judge whether tuning is good or bad. This metric is the optimization direction.
• metric_definitions: regex definitions that pull metric values out of the training logs. The device that lets a custom script's objective metric be recognized.
• Surrogate model: the stand-in model Bayesian search builds to predict "with this combination, performance will be roughly this."
• Multi-fidelity: a method that first sweeps many candidates with few resources (short training) and gives more resources only to the survivors. The core of Hyperband.
• Early stopping: stopping training that is unlikely to improve partway through to save time and cost.
• Warm start: a method that inherits information found by a previous tuning job, so the search starts from good regions instead of a blank slate.