Day 1 - Domains 1 and 2 Integrated Review: Data Preparation + Model Development

📌 Key Takeaways

• Domain 1 (Data Preparation 28%) + Domain 2 (Model Development 26%) = 54% of the exam. Being solid on these two is the foundation of passing.
• One axis connecting the two domains is reproducibility and consistency — the features used in training must be used identically at inference (to prevent skew), and bias and leakage are caught at the data stage.
• Data: real-time → Kinesis, storage → S3, ETL → Glue, visual preparation → Data Wrangler, querying → Athena, feature consistency → Feature Store, bias and explainability → Clarify.
• Models: mapping problem type → built-in algorithm is the most frequently tested (time series DeepAR, anomaly RCF, tabular XGBoost, recommendation FM, text BlazingText).
• Evaluation metrics are determined by whether the data is imbalanced and by the asymmetry of error costs. On imbalanced data, Accuracy easily becomes the wrong answer.

The Whole Map of Domains 1 and 2

This is the first day of the last week. It's not new concepts; we regroup scattered knowledge into a form you can pull out in the exam room.

[Domain 1 — Data Preparation 28%]
  Ingestion          Storage           Preparation/Transformation   Feature management
  ─────              ─────             ──────────               ──────────
  Kinesis DS   ─┐                      Data Wrangler (visual)     Feature Store
  Firehose     ─┼─▶  S3 (data lake) ─▶ Processing Job (code)  ─▶  ├ Online : low-latency single record
  Batch upload ─┘    + Glue Catalog     Glue ETL / EMR (Spark)      └ Offline: S3, bulk
  Ground Truth        (schema/meta)      Athena (ad-hoc SQL)            │
  (labeling)                               Clarify (pre-training bias check)          │
                                                                         ▼
[Domain 2 — Model Development 26%]                                          Training data
  Train ───────────▶  Tune ───────────▶  Evaluate ───────────▶  Model artifact
  Training Job        AMT                 Choose metrics            model.tar.gz (S3)
  Built-in/framework  Bayesian·Random     Accuracy·F1·AUC             │
  /custom container   Grid·Hyperband      RMSE·MAE·R²                 ▼
  File/Pipe/FastFile  objective_metric    Clarify (post-training bias)    Model Registry
  Spot + checkpoints  Debugger (overfitting/bottleneck rules)                  (→ Domain 3)

The horizontal axis of this map is "the order in which data becomes a model," and the vertical axis is "who is responsible for that stage." Exam questions almost always point to one spot on this map. If you first mark "which box is this story about" while reading the scenario, half of the four options are services from other boxes and get eliminated immediately.

Domain 1: Ingestion and Storage — Service Map

First, let's organize where data comes from and where it accumulates.

The exam separates services by "real-time or batch," "write code or do it visually," and "where to store it." Real-time events mean Kinesis, transformation after loading means Glue, visual preprocessing and quick EDA means Data Wrangler, and ad-hoc SQL on S3 means Athena. The center of a data lake is almost always S3, and the rest share the roles of ingestion, transformation, and querying around it.

Domain 1: Choosing Preparation and Transformation Tools

Even the same "preprocessing" splits across tools. This distinction becomes the clue in a question.

⚠️ Pitfall: When it says "visually, writing almost no code," it's Data Wrangler; "run my own preprocessing script as managed," it's Processing Job. Options that flip the two are easy to use as wrong answers. Data Wrangler and Processing Job are not opposed — exporting a Data Wrangler flow to a Processing Job and putting it in a pipeline is the standard combination.

Domain 1: Feature Engineering Checklist

⚠️ Pitfall: Data leakage quietly inflates performance. Two representative cases. ① If you compute the scaler or imputation values on the whole dataset and then split into train/validation, validation information leaks into training — always fit on the training set and only transform the validation set. ② If you shuffle a time series randomly and split it, the future teaches the past — a time series must always be split in chronological order.

Domain 1: Feature Store — Online vs Offline

Feature Store is a central repository where you define a feature once so that training and inference use it together.

import boto3
 
# Online store: single-record lookup right before real-time inference
fs_runtime = boto3.client('sagemaker-featurestore-runtime')
resp = fs_runtime.get_record(
    FeatureGroupName='customer-features',
    RecordIdentifierValueAsString='C-10293',
)
# The offline store accumulates in S3, so query it with SQL via Athena to build a training dataset

The reason Feature Store exists is eliminating training-serving skew. If the features computed at training time and those computed at inference time differ subtly, performance quietly drops. If you define and version features in one place and both online and offline share the same definition, this mismatch disappears. When keywords like "multiple teams and models reuse the same features" or "match training and real-time inference features" appear, it's Feature Store.

Domain 1: SageMaker Clarify — Bias and Explainability

Clarify does two things. ① Bias detection — it measures bias in the data before training and bias in the model after training. ② Explainability — it shows with SHAP values how much each feature contributed to a prediction.

Bias is examined at two points. Pre-training bias measures the imbalance of the data itself (for example, too few samples of a particular group), and post-training bias measures whether the model's predictions come out differently by group. "Check fairness, bias, discrimination" is Clarify; "explain the basis of predictions / feature contribution" is Clarify's SHAP. At the monitoring stage, Model Monitor works with Clarify to watch for bias and feature-attribution drift in production.

Domain 2: Built-in Algorithms → Problem Type Mapping

Model development is "which algorithm, and how do you train it." The representative mapping of built-in algorithms is as follows.

They ask about algorithm mapping. Choosing regression for "multivariate time series forecasting" is wrong — it's DeepAR. "Real-time stream outliers" is RCF, and "sparse recommendation with thousands of dimensions" is Factorization Machines. They also ask about the training data input mode. If the data is larger than the instance disk or you want to reduce startup delay, stream from S3 in Pipe mode (or the newer Fast File mode); if the data is small, copy it all in File mode.

Domain 2: Training Infrastructure — Input Mode, Instances, Spot

Spot must always be paired with checkpoints. Turning on Spot without checkpoint_s3_uri retrains from scratch every time it's reclaimed, which actually loses money. And max_wait >= max_run must hold — because Spot needs extra time to wait for capacity. "I switched to GPU but it didn't get faster" is usually a case where I/O is the bottleneck rather than compute, and an option that attaches a GPU to a tree-based algorithm (XGBoost) is a trap.

Domain 2: Hyperparameter Tuning (AMT)

SageMaker Automatic Model Tuning (AMT) automates the hyperparameter search.

from sagemaker.tuner import HyperparameterTuner, IntegerParameter, ContinuousParameter
 
tuner = HyperparameterTuner(
    estimator=estimator,
    objective_metric_name='validation:auc',   # metric to optimize
    objective_type='Maximize',
    hyperparameter_ranges={
        'max_depth': IntegerParameter(3, 10),
        'eta': ContinuousParameter(0.01, 0.3),
    },
    max_jobs=30,            # total number of training attempts
    max_parallel_jobs=3,    # concurrent runs (smaller is better for Bayesian)
    strategy='Bayesian',
)
tuner.fit({'train': train_input, 'validation': val_input})

Choosing a tuning strategy is "budget efficiency vs parallelism." Bayesian finds good values with few trials, but it refers to earlier results, so its advantage shrinks if you make the parallelism too large; Random/Grid are easy to parallelize but inefficient; and Hyperband cuts hopeless training early to reduce the cost of large-scale searches. "Efficient search with few training runs" is Bayesian, and "save resources via early stopping" is Hyperband as the keyword.

Domain 2: Evaluation — Choosing the Right Metric

Metric selection is almost always decided by "is the data imbalanced, and what do you weigh more."

Choosing Accuracy on imbalanced data is wrong. In fraud detection where 99% are normal, predicting "all normal" gives 99% accuracy but catches no fraud at all. Such scenarios must be evaluated with F1, AUC, and Recall. "Must not miss fraud/disease" is Recall, and "false-alarm cost is high" is Precision as the keyword. Signs of overfitting (training accuracy ↑, validation accuracy ↓) are handled with regularization, dropout, early stopping, and data augmentation.

Easily Confused Pairs in Domains 1 and 2

Symptom → Cause → Action

Tomorrow we'll group Domains 3 and 4 — deployment and orchestration, and monitoring and security — the same way.

📖 Glossary

• Domain weighting: The proportions in which MLA-C01 is tested: Data Preparation 28%, Model Development 26%, Deployment 22%, Monitoring and Security 24%. Domains 1 and 2 are over half.
• training-serving skew: The phenomenon where features built at training time and at inference time differ subtly, so performance quietly drops.
• Feature Store online/offline store: A low-latency single-record lookup store for real-time inference / an S3-based bulk lookup store for training and batch.
• Data leakage: Information that shouldn't be learned (validation data statistics, future values, columns hinting at the answer) leaking into training so that performance is overestimated.
• Processing Job: A job that runs your preprocessing/evaluation script on SageMaker managed instances. The instance shuts down when it finishes.
• SHAP: An explanation technique that shows numerically how much each feature contributed to an individual prediction. Clarify computes this for you.
• Pre-training bias / post-training bias: Whether the data itself is skewed toward a particular group / whether model predictions come out differently by group.
• AMT (Automatic Model Tuning): A SageMaker feature that trains multiple times with different hyperparameter combinations automatically to find the optimum. The objective_metric being optimized must be emitted to the training logs for the tuner to read it.
• Imbalanced data: Data where one class is overwhelmingly more numerous. Accuracy becomes meaningless and F1, AUC, and Recall become necessary.