Day 2 - ML Problem Types and Evaluation Metrics Basics

A new project arrives: "Predict if customers will churn." The first thing an ML engineer must do is not write code, but identify what kind of problem this is. Whether it's a classification or regression problem, whether labels exist or not — all of this determines which algorithms and evaluation metrics are applicable. Misidentifying the problem type makes every subsequent choice wrong.

Today, we'll classify ML problems by learning approach (supervised/unsupervised/reinforcement) and output form (classification/regression/clustering), and explore the core metrics for evaluating classification models — accuracy, precision, recall, F1, and AUC — what each measures, and when to use which metric. This is core to MLA-C01 Domain 2 (Model Development).

Learning Approaches: Do You Have Labels? Do You Have Rewards?

ML algorithms split three ways depending on "what they learn from."

The criterion is simple: If you have labels → supervised. No labels → unsupervised. Trial-and-error to maximize reward → reinforcement. "Predicting customer churn" has historical churn labels, so it's supervised learning. "Grouping customers into similar segments" has no correct groups, so it's unsupervised (clustering). "A game agent maximizes score" is reinforcement learning.

💡 Related Theory: Supervised learning is the problem of approximating a function f(X)=Y from input X to output Y. Because labeling is expensive, semi-supervised learning (learning with few labels) and self-supervised learning (generating labels from data itself, the approach used in large language models) have become important in practice. AWS SageMaker Ground Truth automates and outsources labeling precisely because of this label cost problem.

Output Form: Classification vs Regression vs Clustering

Supervised learning splits again based on what the output is.

• Classification: Output is discrete categories. "Churn/Retain," "Spam/Normal," "Dog/Cat/Bird"
• Regression: Output is continuous numbers. "Tomorrow's revenue," "House price," "Delivery time"

The representative unsupervised approach is Clustering, which groups data without labels into similar clusters (e.g., customer segmentation).

# Classification: XGBoost for churn prediction (output = probability → discrete label)
estimator = sagemaker.estimator.Estimator(
    image_uri=xgboost_image, role=role,
    instance_count=1, instance_type="ml.m5.xlarge",
    hyperparameters={
        "objective": "binary:logistic",   # Binary classification
        "num_round": 100, "max_depth": 5,
    },
)
 
# If it were regression, just change the objective
#   "objective": "reg:squarederror"     # Continuous value prediction

Even with the same XGBoost, a single hyperparameter objective determines whether it does classification or regression. Misidentifying the problem type causes divergence right here.

🔍 Deeper Dive: Even in classification, you must distinguish whether the output is "probability" or "label." binary:logistic returns probability 0~1, and converting to a label requires a threshold. Default is 0.5, but if losing churning customers is unacceptable, lower the threshold to classify more people as "churn risk." Threshold adjustment is exactly the core of the precision-recall tradeoff we'll see next.

Classification Evaluation Starting Point: Confusion Matrix

To evaluate a classification model, you first need to understand the confusion matrix. It's a 2×2 table crossing predictions and actuals.

                 Actual Positive   Actual Negative
Predicted Positive  TP (True Positive)   FP (False Positive)
Predicted Negative  FN (False Negative)  TN (True Negative)

For the "churn prediction" example: TP = correctly identifying someone who churned, FP = incorrectly marking someone who stays as churning, FN = missing someone who actually churned, TN = correctly marking someone who stays as staying. All metrics are combinations of these four cells.

The Accuracy Trap and Precision/Recall

Accuracy = (TP+TN) / Total. Most intuitive but most dangerous. In data where fraud is 0.1%, a model that says "all normal" scores 99.9% accuracy. This is the accuracy trap in imbalanced data.

That's why two metrics focused on the positive class are needed.

• Precision = TP / (TP+FP): "Of what we called positive, what fraction is truly positive?" Use when you want to reduce false alarms (FP).
• Recall = TP / (TP+FN): "Of truly positive cases, what fraction did we catch?" Use when you want to reduce misses (FN).

These have a tradeoff relationship. Lower the threshold → predict more positives → recall↑ precision↓. Raise it → opposite.

📚 Case Study: COMPAS was a recidivism prediction system used by U.S. courts. A 2016 ProPublica investigation revealed that for Black defendants, the false positive rate of classifying them as "high recidivism risk" was roughly 2x higher than for White defendants. Overall accuracy was similar across races, but FP/FN distributions differed by demographic. Looking at a single accuracy number misses fairness issues that you catch when examining the confusion matrix by group.

F1 and AUC: Summarizing to a Single Number

When you care about both precision and recall, use the F1 score, the harmonic mean of both.

F1 = 2 * (Precision * Recall) / (Precision + Recall)

The harmonic mean is used because if either value is low, F1 drops sharply. Precision 0.9 + Recall 0.1 has arithmetic mean 0.5 but F1 of 0.18. It enforces "being good at just one isn't enough."

AUC (Area Under the ROC Curve) measures a model's discriminative power independent of threshold. The ROC curve plots recall (TPR) and false positive rate (FPR) at every threshold; AUC is the area beneath it.

• AUC = 1.0: Perfect classification
• AUC = 0.5: Same as random guessing
• AUC < 0.5: Predicting backwards

💡 Related Theory: AUC can also be interpreted as "the probability that if you randomly sample one positive and one negative example, the model gives the positive a higher score." Since it measures a model's ranking ability before a threshold is set, it's useful in model comparison stages before thresholds are determined. However, on extremely imbalanced data, AUC can appear optimistically high, so PR-AUC (Precision-Recall curve) is also examined.

Regression Evaluation Metrics

Regression works with continuous values, so there's no confusion matrix. Instead, "how close is the prediction to the actual?" is measured.

• MAE (Mean Absolute Error): Average of absolute errors. Less sensitive to outliers.
• MSE / RMSE (Mean Squared Error / Square Root): Squares errors, so sensitive to large errors. RMSE has the same units as the original, making interpretation easier.
• R² (Coefficient of Determination): How much variance does the model explain? Closer to 1 is better.

For problems where large errors must be avoided (e.g., inventory forecasting), use RMSE. For robustness against outliers, use MAE.

Summary

Two key takeaways today. First, ML problems divide by learning approach (supervised/unsupervised/reinforcement) and output form (classification/regression/clustering), and problem identification is the starting point for algorithm and metric selection. Second, don't trust a single accuracy number; examine precision and recall from the confusion matrix, and choose metrics (F1, AUC) that match your business context.

Next, we'll survey the tools that actually solve these problems — AWS's entire ML stack.