Day 3 - Hyperparameter Tuning (AMT): Bayesian·Random·Grid, Early Stopping, Warm Start

📌 Key Takeaways

• AMT automatically runs multiple Training Jobs from an objective metric + hyperparameter ranges + a search strategy to find the best combination.
• Search strategies: the default is Bayesian (reflects previous results, sequentially dependent). Independent parallel is Random, exhaustive is Grid (categorical), and early stopping of long deep learning is Hyperband.
• Early stopping (early_stopping_type='Auto') cuts off hopeless training midway to reduce compute cost.
• Warm start reuses previous tuning results as a starting point — IDENTICAL_DATA_AND_ALGORITHM for identical conditions, TRANSFER_LEARNING when the data has changed.
• Bayesian benefits from sequential learning, so raising max_parallel_jobs too high actually lowers efficiency.

What Is Hyperparameter Tuning

The algorithm you picked yesterday has knobs such as max_depth, eta, and num_round. With dozens or hundreds of combinations it is hard to try them by hand, and SageMaker Automatic Model Tuning (AMT) automates this search. It automatically launches multiple training jobs, trains each with a different combination, and finds the combination with the best objective metric.

There are three key components.

• Objective metric: The target to maximize/minimize (e.g., maximize validation:accuracy, minimize validation:rmse).
• Hyperparameter ranges: The range of each parameter to search (continuous/integer/categorical).
• Search strategy: How to search within the ranges (Bayesian/Random/Grid/Hyperband).

[Hyperparameter ranges] + [Objective metric] + [Search strategy]
            │
            ▼
  ① Select next combination  ◀─────┐
            │                      │
            ▼                      │ ④ Feed the result back to the strategy
  ② Run Training Job               │    (the core of the Bayesian family)
    (max_parallel_jobs at a time)  │
            │                      │
            ▼                      │
  ③ Emit objective metric ─────────┘
            │
            ├─ Lagging training is stopped by early stopping
            │
            └─ max_jobs exhausted → BestTrainingJob is fixed
                        │
                        ▼
        (Warm start) Reused as the starting point of the next tuning job

from sagemaker.tuner import (
    HyperparameterTuner, ContinuousParameter, IntegerParameter
)
 
tuner = HyperparameterTuner(
    estimator=estimator,
    objective_metric_name='validation:auc',  # objective metric
    objective_type='Maximize',                # maximize
    hyperparameter_ranges={
        'eta': ContinuousParameter(0.01, 0.3),     # continuous range
        'max_depth': IntegerParameter(3, 10),       # integer range
        'num_round': IntegerParameter(50, 300),
    },
    max_jobs=20,                  # total number of training jobs
    max_parallel_jobs=3,          # number running concurrently
    strategy='Bayesian',          # search strategy
)
 
tuner.fit({'train': train_s3, 'validation': val_s3})

There are three parameter types for specifying ranges, and which type you use even affects the search method.

For values where the order of magnitude matters, like a learning rate, the difference between 0.001 and 0.01 is far larger than the difference between 0.1 and 0.11. So such parameters are better searched on a log scale rather than a linear one. AMT lets you specify the scale with scaling_type, and the default Auto looks at the range and chooses the appropriate one. Also remember that if you set the range too wide the search scatters, and if too narrow the optimum stays outside the range.

Hyperparameter tuning is a "black-box optimization" problem. You put in inputs (hyperparameters) and get an output (validation performance), but you do not know the formula of the function in between, and a single evaluation (= one training run) is expensive. So rather than blindly trying everything, a strategy that smartly finds good regions with few evaluations is important. This is why Bayesian optimization appears.

Search Strategies: Bayesian·Random·Grid·Hyperband

The exam requires you to distinguish the differences among search strategies. Distinguish four.

Bayesian is the key. From the results of the trials so far, it builds a probabilistic model of "which region is promising" and picks the next combination to try where the information value is highest. In other words, it reflects earlier training results in the next choice, so it finds good values faster with the same budget. Because of this sequential dependency, setting concurrent execution (max_parallel_jobs) too high reduces the benefit.

• Random: Each trial is independent, so parallelizing is easy, but it relies on luck.
• Grid: It looks at every combination so it is thorough, but combinations explode as parameters grow, and in SageMaker it can be used only with categorical parameters.
• Hyperband: It stops poorly performing configurations early during training and directs resources to promising ones, so it is especially efficient for deep learning tuning where training takes long.

The strategy ultimately splits on three conditions: budget, parallelism, and training length.

⚠️ Pitfall: The default answer to "I want to efficiently find good hyperparameters with a limited budget" is Bayesian. However, if it is "try as many as possible quickly in parallel without depending on previous trial results", Random fits better, and if "like deep learning, training is long and I want to save resources by cutting bad configurations early", Hyperband fits better. The strategy choice splits on budget, parallelism, and training length.

Early Stopping

Tuning runs training jobs many times, so it is costly. Early stopping is a key feature for reducing this. If, during training, the objective metric is judged hopeless compared with other trials, that training job is cut off midway.

tuner = HyperparameterTuner(
    estimator=estimator,
    objective_metric_name='validation:auc',
    objective_type='Maximize',
    hyperparameter_ranges=ranges,
    early_stopping_type='Auto',   # enable early stopping
    ...
)

When enabled with early_stopping_type='Auto', SageMaker looks at the intermediate metrics of in-progress training and cuts off jobs that are clearly lagging, saving compute time. This works only if the algorithm emits the metric at every epoch/round (most built-ins support this).

Early stopping touches on the resource allocation problem of "exploration-exploitation". Spending the same resources on every trial is wasteful. It is reasonable to give up quickly on hopeless candidates (stop them) and concentrate resources on promising ones. The Hyperband strategy itself is an active implementation of this idea, and early stopping in ordinary tuning is a lighter version of the same philosophy.

Warm Start

Sometimes you have already run a tuning job once, and now the data has grown a little or you want to widen the ranges slightly and tune again. Searching from scratch each time throws away the knowledge gained earlier. Warm start continues a new tuning by using the results of a previous tuning job as the starting point.

from sagemaker.tuner import WarmStartConfig, WarmStartTypes
 
warm_config = WarmStartConfig(
    warm_start_type=WarmStartTypes.IDENTICAL_DATA_AND_ALGORITHM,
    parents={'previous-tuning-job-name'},   # parent tuning job
)
 
tuner = HyperparameterTuner(
    estimator=estimator, objective_metric_name='validation:auc',
    objective_type='Maximize', hyperparameter_ranges=ranges,
    warm_start_config=warm_config,
)

There are two warm start types.

When tuning ends, retrieve the best combination from the job description.

import boto3
 
sm = boto3.client('sagemaker')
desc = sm.describe_hyper_parameter_tuning_job(
    HyperParameterTuningJobName='my-tuning-job'
)
 
best = desc['BestTrainingJob']
print(best['TrainingJobName'])                              # best-performing training job
print(best['FinalHyperParameterTuningJobObjectiveMetric'])  # metric name and value
print(best['TunedHyperParameters'])                         # best hyperparameter combination
print(desc['TrainingJobStatusCounters'])                    # completed·failed·stopped counts

By looking at the number of stopped jobs in TrainingJobStatusCounters, you can check how much early stopping actually intervened.

⚠️ Pitfall: If you "want to use the results of a previous tuning job to make a new tuning converge faster", it is warm start. If the data and algorithm are unchanged, choose IDENTICAL_DATA_AND_ALGORITHM; if data was added/changed, choose TRANSFER_LEARNING. Warm start shines especially when combined with the Bayesian strategy — because previous trials enter the probabilistic model as prior knowledge.

When Tuning Does Not Meet Expectations: Symptom → Cause → Action

With tuning, the situation of "it runs but the results are poor" is common, and the exam presents that situation as a question stem.

The last row is especially confusing. Because tuning picks combinations to maximize the validation metric, it can overfit to the validation set itself. To trust the tuning result, a final check on data never used in tuning is required. The cause of a scenario like "the validation score rose greatly after tuning, but performance dropped when deployed" is usually this.

In the next post, we look at JumpStart and transfer learning, which use pre-trained models instead of training from scratch, and at training cost optimization.

📖 Glossary

• Hyperparameter: A setting value a human decides before training starts. Tree depth, learning rate, number of iterations, etc.
• AMT (Automatic Model Tuning): A SageMaker feature that automatically runs multiple training jobs to find the best hyperparameter combination.
• Objective metric: The reference score that tuning maximizes or minimizes. For example, validation AUC or RMSE.
• Bayesian optimization: A strategy that, in a wide search space, estimates "where is promising" from results so far to pick the next trial. It finds good values with few runs.
• Hyperband: A strategy that trains many configurations a little, quickly eliminates the poor performers, and concentrates resources.
• Early stopping: A feature that stops in-progress training that looks hopeless rather than waiting until the end.
• Warm start: A feature that inherits the results of a previous tuning job and continues a new tuning from that point.
• max_jobs / max_parallel_jobs: The total number of training jobs tuning will run / how many of them run at the same time.
• Scaling type: A setting that decides whether to sweep the range linearly or on a log scale. Used for values like learning rate where order of magnitude matters.
• Validation-set overfitting: The phenomenon where, after repeated optimization against the same validation data, the score improves only on that data.