Cert Notes/ Commute Study Notes
Roadmap
KOEN
CLF-C02 · FoundationalCloud Practitioner - Foundational
DVA-C02 · AssociateDeveloper - Associate
SAA-C03 · AssociateSolutions Architect - Associate
SOA-C02 · AssociateCloudOps Engineer - Associate
SAP-C02 · ProfessionalSolutions Architect - Professional
DOP-C02 · ProfessionalDevOps Engineer - Professional
SCS-C03 · SpecialtySecurity - Specialty
MLA-C01 · AssociateMachine Learning Engineer - Associate
  • Week 1
    • 1.ML Lifecycle and the Role of ML Engineers
    • 2.ML Problem Types and Evaluation Metrics Basics
    • 3.Day 3
    • 4.SageMaker Overview: Studio, Training/Inference, Built-in Algorithms
    • 5.Week 1 Comprehensive Review — ML Fundamentals & AWS Stack
  • Week 2
    • 1.Data Collection: S3 Data Lake, Kinesis, Batch Ingestion, Data Formats
    • 2.Data Catalog & ETL: AWS Glue and DataBrew
    • 3.Query & Exploration: Athena, Redshift, EDA Basics
    • 4.Data Storage Strategy: Partitioning, Format Optimization, Training Readiness
    • 5.Week 2 Comprehensive Review — Data Collection & Storage Recap
  • Week 3
    • 1.Feature Engineering: The Art of Transforming Data into Numbers Models Can Read
    • 2.SageMaker Data Wrangler: No-Code Data Preparation
    • 3.SageMaker Feature Store: Managing Features as Assets
    • 4.Data Bias·Quality: Clarify, Class Imbalance Handling, Data Split
    • 5.Week 3 Comprehensive Review — Feature Engineering·Data Quality
  • Week 4
    • 1.SageMaker Training Job: Estimator, Input Channels, Instances, Spot
    • 2.Built-in Algorithms: XGBoost, Linear Learner, Image·Text, Input Formats
    • 3.Hyperparameter Tuning (AMT): Bayesian·Random·Grid, Early Stopping, Warm Start
    • 4.JumpStart·Pre-trained Models·Transfer Learning + Training Cost Optimization
    • 5.Week 4 Comprehensive Review: Model Development 1 — SageMaker Training
  • Week 5
    • 1.Custom Training: Script Mode, BYOC, Framework Containers
    • 2.Distributed Training: Data Parallel and Model Parallel
    • 3.Debugging and Profiling: SageMaker Debugger and Profiler
    • 4.Model Evaluation: Metric Selection, Overfitting, Cross-validation, Confusion Matrix
    • 5.Week 5 Comprehensive: Model Development 2 Review
  • Week 6
    • 1.Inference Options Overview: 4 Deployment Modes and Selection Criteria
    • 2.Real-time Endpoints: Configuration, Auto-scaling, Instance Selection
    • 3.Cost & Advanced Inference: Multi-model, Multi-container, Inference Pipeline, Inferentia
    • 4.Batch & Serverless Inference Deep Dive: Throughput Tuning & Cost Tradeoffs
    • 5.Week 6 Synthesis: Inference Deployment Review
  • Week 7
    • 1.Declare parameters, injectible at execution
    • 2.Create package group bundling models of same purpose
    • 3.buildspec.yml — commands CodeBuild executes
    • 4.CloudFormation: SageMaker endpoint declared as code
    • 5.Synthesis code: pipeline end with condition passes → approval triggers deploy
  • Week 8
    • 1.SageMaker Model Monitor: Data Quality and Model Quality Drift
    • 2.Bias and Explainability Drift: Monitoring During Operations with Clarify
    • 3.Filter recent errors from log group
    • 4.Day 4
    • 5.Best practice: aggregating operational metrics + model monitor metrics on one dashboard
  • Week 9
    • 1.Day 1
    • 2.Day 2
    • 3.Data and Model Protection: KMS Encryption and Secrets
    • 4.Day 4
    • 5.Day 5
  • Week 10
    • 1.Day 1
    • 2.Day 2
    • 3.Day 3
    • 4.Day 4
    • 5.Day 5
AIF-C01 · FoundationalAI Practitioner - Foundational
DEA-C01 · AssociateData Engineer - Associate
MLS-C01 · SpecialtyMachine Learning - Specialty
← MLA-C01/Week 5/Day 4
MLA-C01· AssociateWeek 5 · Day 4~14 min read

Day 4 - Model Evaluation: Metric Selection, Overfitting, Cross-validation, Confusion Matrix

Training ends doesn't mean model is good. Judging by one number "95% accuracy" is risky—on data where fraud is 1%, saying "all normal" yields 99% accuracy. Model evaluation's essence is measuring "generalization performance" with "problem-fitting metrics".

In the MLA-C01 exam, evaluation appears as "which metric on imbalanced data", "how to diagnose and mitigate overfitting", "compute precision/recall from confusion matrix". Today we cover metric selection, overfitting/underfitting, cross-validation, confusion matrix—four axes.

Confusion Matrix and Classification Metrics

Classification evaluation starts with the confusion matrix

The rest is Pro only

Week 1 is free for everyone. Week 2 onwards — plus mock exams and unlimited review — is included in the Pro plan.

See ProLogin
PreviousDebugging and Profiling: SageMaker Debugger and ProfilerWeek 5 · Day 3Next Week 5 Comprehensive: Model Development 2 ReviewWeek 5 · Day 5