Day 2 - ML Problem Types and Evaluation Metric Basics

📌 Key Takeaways

• ML problems divide by learning style (supervised/unsupervised/reinforcement) and output form (classification/regression/clustering). Identifying which one you have is the starting point for every later choice.
• Identification rule: labels present means supervised, no labels means unsupervised, learning from rewards means reinforcement.
• Classification evaluation starts from the confusion matrix (TP/FP/FN/TN). Looking at accuracy as a single number leads to wrong judgments.
• Accuracy is a trap on imbalanced data: with 0.1% fraud, "all normal" also scores 99.9%. Look at precision, recall, F1, and PR-AUC.
• The business decides the metric: if a miss is fatal, use recall; if false alarms are costly, use precision.

Learning Style: Are There Labels, or Are There Rewards

When a new project hands you the request "predict whether customers will churn," the first thing an ML engineer should do is not write code but identify what kind of problem this is. If you pick the wrong problem type, every later choice of algorithm and metric goes off track.

ML algorithms fall into three branches based on "what they learn from."

Identification is simple.

• "Predict customer churn" → past churn outcomes (labels) exist → supervised learning
• "Group customers into similar clusters" → there are no ground-truth groups → unsupervised (clustering)
• "A game agent maximizes its score" → trial and error plus rewards → reinforcement learning

Supervised learning is the problem of approximating a function f(X)=Y from input X to output Y. Because labeling is expensive, in practice semi-supervised learning (training with few labels) and self-supervised learning (creating labels from the data itself, the way large language models are trained) have become important. This labeling cost is why Amazon SageMaker Ground Truth exists as a service that automates and outsources labeling.

Output Form: Classification vs Regression vs Clustering

Supervised learning splits again in two depending on what the output is.

# Classification: predicting churn with XGBoost (output = probability → discrete label)
estimator = sagemaker.estimator.Estimator(
    image_uri=xgboost_image, role=role,
    instance_count=1, instance_type="ml.m5.xlarge",
    hyperparameters={
        "objective": "binary:logistic",   # binary classification
        "num_round": 100, "max_depth": 5,
    },
)
 
# For regression, you only change the objective
#   "objective": "reg:squarederror"     # continuous value prediction

Even with the same XGBoost, a single objective hyperparameter separates classification from regression. If you misidentify the problem type, things go wrong from here.

Even within classification, you must distinguish whether the output is a "probability" or a "label." binary:logistic returns a probability from 0 to 1, and converting that to a label requires a threshold. The default is 0.5, but if the business can't afford to miss churners, you lower the threshold to classify more people as "churn risk." Threshold adjustment is the heart of the precision-recall trade-off we look at next.

The Starting Point of Classification Evaluation: The Confusion Matrix

To evaluate a classification model, you first need to understand the confusion matrix. It is a 2x2 table crossing predictions with actuals.

                 Actual Positive       Actual Negative
Predicted Pos.   TP (true positive)    FP (false positive)
Predicted Neg.   FN (false negative)   TN (true negative)

              ┌──────────── Precision = TP / (TP + FP)  ← based on the predicted-positive row
              │
   TP  FP ────┘
   FN  TN ────┐
              │
              └──────────── Recall = TP / (TP + FN)  ← based on the actual-positive column

Using the "churn prediction" example, you read it like this.

Every metric is a combination of these four cells.

The Accuracy Trap and Precision/Recall

Accuracy = (TP+TN) / total. The most intuitive, and the most dangerous. On data where fraud is 0.1%, even a model that guesses "all normal" gets 99.9% accuracy. This is called the accuracy trap on imbalanced data.

So we need two metrics focused on the positive class.

• Precision = TP / (TP+FP): "Of what it called positive, the share that is truly positive." Use it when you want fewer false alarms (FP).
• Recall = TP / (TP+FN): "Of the truly positive cases, the share it caught." Use it when you want fewer misses (FN).

The two are in a trade-off. Lowering the threshold predicts more positives, so recall goes up and precision goes down; raising it does the opposite.

One-line rule for choosing a metric: prioritize the metric that reduces whichever of FN and FP is more expensive for the business.

COMPAS is a recidivism prediction system that US courts used. A 2016 ProPublica report revealed that the rate of false positives (FP), wrongly classifying defendants as "high risk of reoffending," was about 2x higher for Black defendants than for white defendants. Overall accuracy was similar across races, but the distribution of FP/FN differed by group. The lesson: a fairness problem that a single accuracy number misses can only be caught by looking at the confusion matrix level.

F1 and AUC: Summarizing in a Single Number

When you care about both precision and recall, you use the F1 score. It is their harmonic mean.

F1 = 2 * (Precision * Recall) / (Precision + Recall)

The reason for a harmonic mean is that F1 drops sharply if either one is low. With precision 0.9 and recall 0.1, the arithmetic mean is 0.5 but F1 is 0.18. It enforces "being good on only one side isn't enough."

AUC (area under the ROC curve) measures a model's discriminating power regardless of threshold. The ROC curve plots recall (TPR) against the false positive rate (FPR) at every threshold, and the area under it is the AUC.

Which metric to look at when: this table applies directly to exam identification questions.

AUC can also be interpreted as "the probability that, when you randomly pick one positive sample and one negative sample, the model gives the positive a higher score." Because it looks at the model's own ranking ability before a threshold is set, it is useful at the model comparison stage when the threshold isn't decided yet. However, on extremely imbalanced data AUC can come out optimistic, so also look at PR-AUC (the precision-recall curve).

Regression Evaluation Metrics

Regression deals with continuous values, so there is no confusion matrix; it measures "how close the prediction is to the actual."

• For problems where large errors must be especially avoided (inventory forecasting, etc.), use RMSE.
• If there are many outliers and you need robustness, use MAE.
• When comparing models across different datasets, the unitless R² is convenient.

⚠️ Pitfall: RMSE and MAE share the property that "lower is better," but you must not compare their magnitudes directly. On the same data RMSE is always at least MAE, and a large gap between them is a signal that a few large errors (outliers) exist.

Confusion Matrix in Numbers

Calculating once sticks longer than memorizing metrics. Suppose a fraud detection model applied to 1,000 transactions gives the following.

                 Actual fraud (30)    Actual normal (970)
Predicted fraud   TP = 18              FP = 42
Predicted normal  FN = 12              TN = 928

• Accuracy of 0.946 is even lower than the model that guesses "all normal" (0.970). On imbalanced data, accuracy gives an illusion that can't even clear the baseline.
• Lowering the threshold increases both TP and FP, moving toward higher Recall and lower Precision.
• If the loss from one fraud case is much larger than the handling cost of one false alarm, raising Recall is the right direction.

Averaging Methods for Multiclass Classification

With three or more classes, you compute precision and recall per class and then average. The averaging method changes the conclusion.

⚠️ Pitfall: In multiclass classification with imbalanced classes, looking only at the weighted average hides minority-class failures behind majority-class performance. If the goal is "catching the minority class well," check the macro average or per-class metrics separately.

What Happens When You Pick the Wrong Metric

In the next article, we survey the tools that actually solve these problems: AWS's whole ML stack.

📖 Glossary

• Label: The answer attached to data. The value the model must predict, such as "this customer churned."
• Confusion matrix: A 2x2 table crossing predictions with actuals. The four cells TP, FP, FN, and TN produce every classification metric.
• False positive (FP): Said yes when it was no. Sending normal mail to spam.
• False negative (FN): Said no when it was yes. Letting fraud through as normal.
• Precision: Of what was called positive, the share that was actually right. A metric that reduces wasted shots.
• Recall: Of actual positives, the share caught. A metric that reduces misses.
• F1 score: The harmonic mean of precision and recall. If only one side is good, the score plunges.
• Threshold: The cut line that turns a predicted probability into positive/negative. Lowering it raises recall, raising it raises precision.
• ROC-AUC: A 0-1 measure of the "ability to give positives higher scores," independent of threshold. 0.5 is guessing level.
• PR-AUC: The area under the precision-recall curve. More honest than ROC-AUC on imbalanced data where positives are very rare.