Day 5 - Week 1 Comprehensive Review — ML Fundamentals & AWS Stack

📌 Key Takeaways

• ML cycles through prepare → develop → deploy → monitor → retrain. The four exam domains map 1:1 onto this cycle.
• Each problem type (classification, regression, clustering) has different evaluation metrics. With imbalanced data, beware the accuracy trap.
• SageMaker covers the entire lifecycle — training runs in ephemeral containers, and inference is chosen by traffic pattern.
• For common tasks, attach an AI service with no training (Comprehend, Transcribe, Rekognition, Textract, etc.).
• The layer-selection criterion is generality vs. uniqueness of the problem — general means an AI service, unique means SageMaker.

ML Lifecycle: The Full Picture

MLA-C01 does not test isolated facts; it asks "for this situation, which tool at which stage?" The most efficient way to study is to layer concepts and services on top of the lifecycle.

[1. Data Prep] → [2. Model Dev] → [3. Deploy·Serve] → [4. Monitor·Maintain]
  Collect/Store/Transform   Train/Tune/Evaluate   Pick Endpoint   Drift/Retrain
  (Domain 1, 28%)           (Domain 2, 26%)       (Domain 3, 22%) (Domain 4, 24%)
        ↑                                                              |
        └──────────────── Retrain Trigger ──────────────────────────┘

The key is the last arrow. Over time, the data distribution of a deployed model changes (drift) and performance drops, which sends you back to the data preparation stage. An ML engineer's job is to automate and stabilize this cycle.

Collecting what we learned this week by stage into one table gives this:

CRISP-DM (Cross-Industry Standard Process for Data Mining) is the standardization of this cycle. Established in the 1990s, this methodology defines a six-stage iterative cycle: business understanding → data understanding → data preparation → modeling → evaluation → deployment. Modern MLOps is a form that tightly couples this with "post-deployment monitoring and automated retraining." On the exam, a scenario saying "model performance degrades over time" always points to the monitoring and retraining stage of this cycle.

Problem Types and Evaluation Metrics

The evaluation metric depends on what problem the model solves. Choose the wrong metric and you end up building a "99% accurate but useless model."

The learning approaches, summarized again:

• Supervised learning: learns from data with answers (labels) → classification·regression
• Unsupervised learning: finds structure without answers → clustering·dimensionality reduction·anomaly detection
• Reinforcement learning: learns a policy from rewards → sequential decision-making

from sklearn.metrics import precision_score, recall_score, f1_score
 
# With imbalanced data (0.1% fraud), accuracy is a trap — look at precision/recall
precision = precision_score(y_true, y_pred)   # Share of positive predictions that are truly positive
recall = recall_score(y_true, y_pred)         # Share of actual positives that were caught
f1 = f1_score(y_true, y_pred)                 # Harmonic mean of the two

A decision table for choosing a metric — practice naming the metric straight from the situation.

The Precision-Recall tradeoff comes from asymmetric costs. A problem like fraud detection, where "missing a case is a disaster," prioritizes Recall (not missing anything), while a problem like a spam filter, where "blocking a legitimate email is troublesome," prioritizes Precision (reducing false alarms). When the classes are extremely imbalanced, Accuracy is meaningless — guessing the majority class for everything still yields 99%. That is why F1 or AUC-PR is used as the summary metric for imbalanced problems.

⚠️ Gotcha: It is easy to confuse RMSE and MAE in regression. RMSE squares large errors and penalizes them more heavily, so it is sensitive to outliers, while MAE treats all errors equally and is robust. If "there are many outliers and large errors must be specially prevented," choose RMSE; if "you want to reduce the influence of outliers," choose MAE.

SageMaker Core Features Summary

The SageMaker we saw in the second half of this week is a toolset that covers the whole ML lifecycle. Regrouped by stage:

Two points fall within the exam scope: training happens in ephemeral containers (the instance shuts down automatically when it finishes, and Spot reduces cost), and inference options are split by traffic pattern.

import sagemaker
from sagemaker.estimator import Estimator
 
session = sagemaker.Session()
role = sagemaker.get_execution_role()
 
# Train a tabular classification with built-in XGBoost
estimator = Estimator(
    image_uri=sagemaker.image_uris.retrieve("xgboost", session.boto_region_name, "1.7-1"),
    role=role,
    instance_count=1,
    instance_type="ml.m5.xlarge",
    use_spot_instances=True,   # Reduce training cost
)
estimator.fit({"train": "s3://my-bucket/train/"})

One-line summary of choosing an inference option

If the built-in algorithms do not fit, you have two ways to widen the options. Use script mode (an AWS-managed framework container + your own training script) to run TensorFlow or PyTorch models, or, if you need a special runtime, go to a custom container (a Docker image you build yourself). In the order "built-in → script mode → custom container," control grows and so does the burden.

AWS AI Services: Pre-Trained Models Without Training

You do not need to train every ML problem yourself. AWS provides pre-trained, managed AI services as APIs. If the task is "common and I want to attach it quickly," this side is the answer.

The exam also includes scenarios that combine several AI services.

The deciding criterion is the level of abstraction. AI services (API calls, no training needed) → SageMaker (train and deploy yourself, flexibility) → building directly on EC2 (maximum control, maximum burden). "Common problem + fast implementation" means an AI service; "custom model on your own data" means SageMaker.

This hierarchy is the cloud version of "build vs buy." An AI service means borrowing a model AWS pre-trained on massive amounts of data (buy), and SageMaker means building one yourself with your own data (build). The ML engineer's decision point is "is my problem general or unique?" For general NLP and vision tasks, an AI service is faster, more accurate, and carries no operational burden, but domain-specific data (e.g., medical imaging, industrial defects) requires training your own model.

Week 1 One-Line Checklist

Checking Week 1 Through Scenarios

The exam does not ask about concepts separately; it embeds them in situations. Let's apply this week's content in a situation → judgment format.

Commonly Confused Pairs

⚠️ Gotcha: The easy mistake in Week 1 is the habit of choosing the "better technology." The exam treats the simplest and cheapest choice that satisfies the requirements as the correct answer. Choosing deep learning for a problem that does not need it, or custom training for a task a single API call can handle, is a wrong answer.

Next week (Week 2) we dig deep into the first stage of the lifecycle, data collection and storage — S3, Kinesis, Glue, and Athena.

📖 Glossary

• CRISP-DM: A classic data mining methodology that repeats six stages from business understanding to deployment. The ancestor of MLOps.
• Drift: The phenomenon in which the input distribution or the input-answer relationship changes over time and model performance drops.
• Retraining trigger: A mechanism that automatically reruns training when drift or performance degradation is detected.
• Macro-F1: The simple average of per-class F1 in multi-class classification. It reflects minority classes equally.
• Silhouette score: A metric measuring how tightly clusters hold together and how well separated they are. Used when there are no answers.
• R² (coefficient of determination): A regression metric expressing, from 0 to 1, how much of the data's variation the model explains.
• Automatic Model Tuning: A SageMaker feature that automatically searches hyperparameter combinations to find the best values.
• Foundation model: A large model pre-trained on vast data and used broadly across tasks. Bedrock provides it as an API.
• build vs buy: The judgment of whether to build it yourself or buy and use something that already exists. The axis for choosing between AI services and SageMaker.
• Managed service: A service where AWS takes over infrastructure operations. You give up some control in exchange for less burden.