Day 1 - The ML Lifecycle (from a Specialty Perspective)

📌 Key Takeaways

• MLS-C01 asks not "where the SageMaker buttons are" but about trade-offs between stages. It weaves data, features, algorithms, metrics, and deployment into a single scenario.
• When translating a business goal into an ML problem, first fix three things: prediction target, input features, and success metric.
• The same problem changes type with label availability: supervised classification ↔ unsupervised anomaly detection (a trap that is easy to mix up on the Specialty exam).
• The lifecycle is a cycle, not a line. When drift is detected in operation, monitoring triggers retraining and returns to the data stage.
• Offline metrics are a gate, and online business metrics (A/B) are the final verdict. Accuracy alone is meaningless on imbalanced data.

Problem Definition: Translating a Business Question into an ML Problem

The most common failure happens not in modeling but in problem definition. "We want to reduce churn" is a business goal, not an ML problem. To translate it, you must fix three things.

1. Prediction target (target): What are you predicting — whether a customer churns within the next 30 days (binary classification)?
2. Input features (features): What signals do you predict with — recent login frequency, payment history, support tickets?
3. Success metric (metric): What makes a model "good" — maximize precision under the condition recall ≥ 0.8?

If you get the problem type wrong, everything after it goes off the rails. The Specialty exam keeps testing this mapping through scenarios.

Problem Type Mapping Table

Supervised learning learns an input→output mapping from labeled data, and unsupervised learning finds structure without labels. Fraud detection is usually supervised binary classification, but if past fraud cases (positive labels) are extremely scarce, it shifts to unsupervised anomaly detection (Random Cut Forest). The same business problem changes problem type depending on label availability.

⚠️ Pitfall: If you memorize "fraud detection = always classification," you will pick the wrong answer in a scenario for a new service that has no labels. When you find clues in the passage like "there are almost no past cases" or "there is no labeling budget," unsupervised anomaly detection rises to a candidate answer.

The Lifecycle Is a Cycle, Not a Line

An ML system is not built once and done. When data changes during operation (drift), the cycle starts over.

1. Data       : Collect → Clean → Label → Store (data lake)
2. Features   : Feature engineering → Transform → Feature Store
3. Model      : Algorithm selection → Training → HPO tuning → Evaluation
4. Deploy     : Real-time endpoint / Batch transform / Serverless
5. Monitor    : Data & model quality drift → Retraining trigger
                └──────────(loop back to stage 1)──────────┘

• This week (Week 1) focuses on stage 1 (data) and the stages just before it — ingestion, storage, and labeling.
• The Specialty exam gives data engineering a weight of about 20%.
• Separating each stage into its own job makes reproducibility and re-runs easy. You can re-run just the cleaning, or put a different algorithm on top of the same features.

# The lifecycle as seen through the SageMaker SDK — separating concerns by stage
import sagemaker
from sagemaker.processing import ProcessingInput, ProcessingOutput
from sagemaker.sklearn.processing import SKLearnProcessor
 
session = sagemaker.Session()
role = sagemaker.get_execution_role()
 
# Stages 1-2: cleaning + feature engineering as a Processing Job
processor = SKLearnProcessor(
    framework_version="1.2-1",
    role=role,
    instance_type="ml.m5.xlarge",
    instance_count=1,
)
processor.run(
    code="preprocess.py",
    inputs=[ProcessingInput(source="s3://my-lake/raw/", destination="/opt/ml/processing/input")],
    outputs=[ProcessingOutput(source="/opt/ml/processing/train", destination="s3://my-lake/features/train")],
)

Training-serving skew is the performance degradation that happens when the feature transformation logic at training time differs from the logic at inference time. Fixing preprocessing in code (preprocess.py) and reusing it identically for training and inference, or managing features centrally with SageMaker Feature Store, reduces skew. Features improvised ad hoc in a notebook almost always create skew.

Choosing a Deployment Form (Stage 4)

Even the same model gets a different deployment form depending on the nature of the traffic. The Specialty exam tests this choice through scenarios too.

⚠️ Pitfall: If you see only "reduce cost" and pick serverless unconditionally, you are wrong. If there is a clue that response latency (SLA) is strict, serverless with its cold starts is the wrong answer, and a real-time endpoint is the answer.

Data Splitting: Random Is Not Always Right

• General tabular data: Random split (train/validation/test). If classes are imbalanced, use a stratified split to preserve the proportions.
• Time series data: You must use a chronological split. Putting future data into training means using information that could not actually have been known, which is leakage.
• Grouped data (multiple rows from the same user): Split by grouping at the user level. If the same user is scattered across training and validation, performance is inflated.

Connecting Offline Model Metrics to Business Metrics

This is where the Specialty exam digs deepest. A single metric like accuracy deceives people. If fraud is 0.1% of all transactions, predicting only "all legitimate" yields 99.9% accuracy. That is why you must look at class imbalance and error costs together.

# Core classification metrics (based on the confusion matrix)
from sklearn.metrics import precision_score, recall_score, f1_score, roc_auc_score
 
# precision = TP / (TP + FP)  → "of those flagged as fraud, the fraction that is real" (cost of false positives)
# recall    = TP / (TP + FN)  → "of actual fraud, the fraction caught"                  (cost of false negatives)
precision = precision_score(y_true, y_pred)
recall    = recall_score(y_true, y_pred)
f1        = f1_score(y_true, y_pred)          # harmonic mean of precision and recall
auc       = roc_auc_score(y_true, y_score)    # threshold-independent, ranking quality

Metric Interpretation Table — What Each Says in Which Situation

Business context decides which metric to optimize.

• Fraud detection / cancer diagnosis: Missing even one is fatal → prioritize recall
• Spam filter / marketing targeting: False positives are expensive (blocking legitimate mail) → prioritize precision
• Imbalance + threshold undecided: Compare only the model's pure discriminative power with AUC or PR-AUC

ROC-AUC (the area under the TPR-FPR curve) is relatively insensitive to class imbalance. But when positives are extreme, around 0.1%, ROC-AUC looks overly optimistic, so PR-AUC (the area under the precision-recall curve) gives a more honest signal. When the Specialty exam asks "which metric for imbalanced data?", the answer is usually PR-AUC, or whichever of recall/precision has the higher cost.

In a metric-selection question, "training loss" is a wrong answer. Loss is only an optimization target and does not reflect the business cost structure.

Online Validation with A/B Testing

Good offline metrics do not guarantee that actual user behavior (revenue, dwell time) improves. That is why deployment is not a one-time event but a process of splitting traffic to validate. SageMaker puts multiple variants on one endpoint and distributes traffic by weight.

from sagemaker.session import production_variant
 
variant_a = production_variant(model_name="model-v1", instance_type="ml.m5.large",
                               initial_instance_count=1, variant_name="A", initial_weight=90)
variant_b = production_variant(model_name="model-v2", instance_type="ml.m5.large",
                               initial_instance_count=1, variant_name="B", initial_weight=10)
 
session.endpoint_from_production_variants(
    name="fraud-endpoint", production_variants=[variant_a, variant_b]
)
# Send only 10% to the new model (B), compare business metrics in CloudWatch, then adjust weights

[Offline gate]                       [Online verdict]
Train/validation set metrics ──pass──▶ Small-traffic A/B ──improvement confirmed──▶ Increase weight
   (PR-AUC etc.)         │                    │
                       fail                worse
                         ▼                    ▼
                  Do not deploy       Roll back (weight 0)

• Offline metrics are a gate — if they do not pass, you do not deploy.
• Online metrics are the final verdict — the actual business numbers are the answer.
• This separation is the operational sense the Specialty exam demands.

The Order for Solving Scenario Questions

Specialty passages stack multiple constraints. Reading in the order below solves most of them.

1. What is being predicted → fix the problem type (classification/regression/clustering/anomaly detection/recommendation/time series).
2. Are there labels → if not, all supervised-learning candidates are eliminated.
3. Are the classes balanced → if imbalanced, discard the accuracy-family options first.
4. Which is more expensive, false positives or false negatives → this sets the priority between precision and recall.
5. Response latency, data sensitivity, cost constraints → these set the deployment form and operating approach.

In a passage where constraints conflict, the answer is not the flashiest technology but the choice that satisfies all the constraints at once. If it violates even one constraint (for example, sensitive data sent to a public workforce), it is wrong regardless of its other advantages.

Tomorrow we cover where and in what format to keep this data (S3, EFS, FSx for Lustre; RecordIO, Parquet).

📖 Glossary

• Label : The correct answer attached to data. The value the model must predict, such as "this transaction was fraud."
• Class imbalance : A state where the ratio of positives to negatives is extremely skewed. For example, fraud at 0.1%.
• Precision : The fraction of true positives among those called positive. It looks at the cost of false positives.
• Recall : The fraction of actual positives that were caught. It looks at the cost of false negatives.
• PR-AUC : The area under the precision-recall curve. More honest than ROC-AUC under extreme imbalance.
• ROC-AUC : The area of the TPR-FPR curve drawn while varying the threshold. The model's ranking ability.
• Drift : The phenomenon where the input data distribution or the relationship to the correct answer changes in operation and model performance falls.
• Training-serving skew : An incident where the feature computation logic used at training time differs from the logic at inference time, degrading performance.
• Production variant : A model version placed on one endpoint. Traffic is split by weight to run an A/B test.
• Random Cut Forest : SageMaker's built-in unsupervised anomaly detection algorithm. It assigns anomaly scores without labels.