Day 2 - Feature Engineering: Scaling, Encoding, and Binning

📌 Key Takeaways

• Scaling is essential for distance- and gradient-based models (KNN, SVM, linear/logistic regression, neural networks, PCA) and unnecessary for tree-based models.
• When there are many outliers, use RobustScaler (based on median and IQR) instead of Min-Max.
• For encoding: Ordinal if there is an order, One-Hot if there isn't, and Target/Hashing/embeddings for high cardinality.
• Target encoding is powerful but carries the risk of target leakage. It needs K-fold or smoothing.
• Binning gives linear models nonlinearity but you accept information loss. The common principle is to fit every transformation on the training set only.

Scaling: Normalization vs. Standardization

Once cleaning has made the data tidy, it is time to transform it into a form the model can learn from well. Feature engineering carries so much weight in the ML pipeline that it is commonly said to "determine 80% of model performance."

Many algorithms are sensitive to the scale (magnitude) of features. If you feed in age (0 to 100) and income (0 to 100 million) as they are, income overwhelms the distance and gradient calculations.

• Algorithms where scaling is required: distance-based (KNN, K-means), gradient-based (linear/logistic regression, neural networks), models with a regularization term (Ridge/Lasso), SVM, PCA.
• Algorithms where scaling is unnecessary: tree-based (Decision Tree, Random Forest, XGBoost, LightGBM). Trees only look at split thresholds within a single feature, so they are invariant to monotonic transformations.

from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler
 
# Standardization: default for neural networks/linear models
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)   # not fit!
 
# With many outliers, use RobustScaler
robust = RobustScaler()
X_train_robust = robust.fit_transform(X_train)

[When scales differ wildly]     [After standardization]
 Loss function contours are      Contours become nearly circular
 elongated ellipses              ↓
      ╱╲                     ╭─╮
   ╱      ╲   zigzag descent  │ ●│  converges along the shortest path
  ╲      ╱                   ╰─╯
      ╲╱

The reason gradient descent is sensitive to scale is the shape of the loss function's contours. If feature scales differ wildly, the contours become elongated ellipses, and the gradient zigzags slowly toward the minimum. If standardization puts all features on the same scale, the contours become nearly circular and convergence is fast and stable. For distance-based models (KNN, K-means), it prevents Euclidean distance from being dominated by large-scale features.

⚠️ Trap: Just like an imputer, a scaler must be fit on the training set only. Using the test set's min/max/mean is leakage. Also, with Min-Max a single outlier pulls max up and compresses the remaining values toward 0, so when outliers are suspected, RobustScaler is the safe choice.

Encoding: Turning Categorical Values into Numbers

Most algorithms accept only numbers as input, so categorical variables must be converted into numbers.

Encoding Decision Tree

Is there an order?
├─ Yes → Ordinal / Label Encoding  (low < mid < high)
└─ No → How many unique values are there?
             ├─ Few (a few dozen or fewer) → One-Hot
             └─ Many (thousands or more) → Target / Frequency / Hashing / Embedding
                                   · Watch for leakage (Target)
                                   · Accept collisions (Hashing)
                                   · For deep learning, Embedding

The key distinction is whether there is an order. For grades (low < medium < high), Ordinal/Label is natural, and for something with no order like colors (red/blue/green), One-Hot is right. If you Label Encode colors for a linear model (red=0, blue=1, green=2), it creates the false order "green > blue > red."

from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder
 
# Nominal: one-hot
ohe = OneHotEncoder(handle_unknown="ignore", sparse_output=False)
X_color = ohe.fit_transform(df[["color"]])
 
# Ordinal: ordinal with an explicit order
oe = OrdinalEncoder(categories=[["low", "mid", "high"]])
df["level_enc"] = oe.fit_transform(df[["level"]])

Target encoding is powerful but dangerous.

# Target encoding (compute statistics from training-set folds only, to prevent leakage)
means = X_train.groupby("city")["target"].mean()
X_train["city_te"] = X_train["city"].map(means)
X_test["city_te"] = X_test["city"].map(means)   # apply training-set statistics

With high cardinality (for example, thousands of postal codes), target encoding avoids the dimension explosion of one-hot while compressing the relationship with the target into a single column. However, if you build a feature from the target of the same data, target leakage occurs, where the model peeks at the answer. To prevent this, use K-fold target encoding (computing each fold's statistics from the other folds) or smoothing (weighting the overall mean and the category mean by sample count). On the exam, the key is the trade-off: "high cardinality → one-hot explodes dimensions, and target encoding requires care about leakage."

⚠️ Trap: With One-Hot encoding, if the test set contains a new category not seen during training, it raises an error. You must set handle_unknown="ignore" so that unknown categories are all treated as zero vectors, to be safe in production.

If you apply target encoding as-is to a category with extremely few samples (for example, only 3 records for that city), the value swings to extremes. You need to pull it toward the overall mean with smoothing.

Binning: Turning Continuous Values into Intervals

Binning is a technique that splits a continuous variable into intervals (bins) and treats it like a categorical one.

Effects of binning:

• Captures nonlinear relationships: A linear model can learn nonlinear patterns such as "purchase rate jumps between ages 30 and 40" through bin dummies.
• Mitigates outliers and noise: Extreme values are lumped into the same top bin, which reduces their influence.
• Easier to interpret: A "high-income bin" is easier to explain than continuous income.

The downside is information loss. Fine differences within a bin disappear.

import pandas as pd
 
# Equal-width binning
df["age_bin"] = pd.cut(df["age"], bins=[0, 18, 35, 60, 120],
                       labels=["minor", "young", "middle", "senior"])
 
# Equal-frequency binning (quartiles)
df["income_q"] = pd.qcut(df["income"], q=4, labels=["Q1", "Q2", "Q3", "Q4"])

Binning is a tool for adjusting the model's bias-variance trade-off. If you use continuous values as they are, a linear model can only express a monotonic straight-line relationship (high bias), but if you convert them to bin dummies, it learns an independent effect in each bin and captures nonlinearity (reduced bias). However, if you split bins too finely, each bin has few samples and estimates become unstable (increased variance). Equal-frequency binning mitigates this variance problem by equalizing the sample count in each bin.

Applying binning to tree or boosting models gains almost nothing, because trees already find split points on their own and express nonlinearity. In questions that ask "which preprocessing has little effect," the model type is the clue.

Derived Features: Interactions and Polynomials

If scaling, encoding, and binning are tasks that "change the form," derived features are tasks that "create signal that wasn't there."

• Ratio features: Often amount / average amount or debt / income is a stronger signal than the amount itself.
• Interaction features: The product of two features (age × income) expresses a combined effect such as "high-income middle-aged."
• Polynomial features: Adding x², x³ lets a linear model express curved relationships.
• Aggregate features: Attach group statistics, such as each user's average purchase amount over the last 30 days.

Polynomials and interactions grow dimensionality quickly and lead to overfitting. Applying a degree-2 polynomial to 3 features gives 10, and to 10 features gives 66. Use them together with regularization (Ridge/Lasso), or, for tree models, they are often unnecessary in the first place.

Aggregate features become leakage if you pick the wrong aggregation point in time. "This user's average purchase amount over their entire history" can include data from after the prediction time. You must aggregate using only data from before the prediction time.

Feature Selection: More Isn't Better

Once you've created features, you also have to cut them back. Unnecessary features increase overfitting, training time, and difficulty of interpretation all together.

• Multicollinearity: If several features are strongly correlated with each other, coefficient interpretation in linear models becomes unstable. Keep just one or combine them with PCA.
• Constant and quasi-constant columns: A column with practically one value carries no information, so remove it.
• ID columns: If you feed a user ID in as is, the model overfits by memorizing individual rows. Convert it to an embedding or an aggregate.

Preprocessing Requirements by Model

In the next article, we look at feature engineering for special data such as dates and text, and at handling high-cardinality categorical data.

📖 Glossary

• Feature engineering: The work of turning raw data into input features that a model can learn from well.
• Normalization (Min-Max): Scaling that squeezes values into the [0,1] range. Very sensitive to outliers.
• Standardization (Z-score): Scaling that transforms values to mean 0 and variance 1. The default for linear models and neural networks.
• RobustScaler: A method that uses the median and IQR to scale robustly against outliers.
• Invariant to monotonic transformations: The property that the result is the same as long as the ordering of values is preserved. Tree models have this property.
• Nominal / ordinal: Categories with no order (colors) / categories with an order (grades).
• Cardinality: The number of unique values of a categorical variable. Thousands means high cardinality.
• Target Encoding: An encoding that replaces a category with that category's target mean. Watch out for leakage.
• Target leakage: An accident where information about the answer being predicted gets mixed into the features, inflating performance.
• Binning: A technique that groups continuous values into intervals and treats them like categories. You gain nonlinearity and lose information.