Day 5 - Week 3 Comprehensive Review: Cleaning and Feature Engineering

📌 Key Takeaways

• Week 3 covered the first half of EDA — data cleaning and feature engineering. This is the core of MLS-C01 Domain 2.
• One principle runs through the whole week: prevent data leakage. Every transformation is fit on the training set only; everything else just gets transform.
• The second principle is preprocessing matched to the model. Linear models need scaling, one-hot encoding and binning; tree-based models mostly do not.
• The basics: missingness by mechanism (MCAR/MAR/MNAR), outliers by robust statistics (IQR), and duplicates cleaned before the split.
• Tool mapping: visual preparation = Data Wrangler, large-scale code-based processing = Processing Job, feature reuse and consistency = Feature Store.

Flow Map: Cleaning → Feature Engineering → Tools

This week covered the first half of EDA, data cleaning and feature engineering. Both are central to MLS-C01 Domain 2 (Exploratory Data Analysis) and are the stages that most determine model performance in practice.

[Raw Data]
   │
   ├─ 1) Cleaning (Day 1)
   │     Missing values: judge MCAR/MAR/MNAR → impute (mean/median/KNN/model)
   │     Outliers: detect with IQR/Z-score → delete/clip/transform/keep
   │     Duplicates & errors: clean before the split (prevent leakage)
   │
   ├─ 2) Feature Engineering (Day 2–3)
   │     Scaling: normalization/standardization/RobustScaler (not needed for trees)
   │     Encoding: Label/One-Hot/Target (decided by order and cardinality)
   │     Binning: nonlinearity and interpretability ↔ information loss
   │     Dates: component decomposition + sin/cos
   │     Text: BoW → TF-IDF → embeddings
   │     High cardinality: target/hashing/embeddings
   │
   └─ 3) AWS Tools (Day 4)
         Data Wrangler (visual) → Processing Job (scale) → Feature Store (reuse)

Key Decision Points

Missing Values: How Should You Impute?

Scaling: Whether It Is Needed, by Algorithm

• Needed: KNN, K-means, linear/logistic regression, SVM, neural networks, PCA, Ridge/Lasso
• Not needed: Decision Tree, Random Forest, XGBoost, LightGBM (tree-based)
• With many outliers: RobustScaler

Encoding: Decided by Order and Cardinality

Is there an order?  ── Yes ──→ Ordinal/Label Encoding
        │
        └─ No ──→ Are there many unique values?
                        ├─ Few  → One-Hot
                        └─ Many → Target/Frequency/Hashing/Embedding

Outliers: Detection and Treatment Are Separate

💡 Concept: One principle runs through this entire week: preventing data leakage. Imputation statistics (mean, median), scaler parameters (min/max/μ/σ), target-encoding statistics, the vocabulary of a text vectorizer — every one of these transformations must be fit on the training data only, and validation and test data get only transform. Duplicate removal belongs before the split, and time-series features must respect point-in-time. Leakage inflates offline performance unrealistically and then collapses in production; it is a trap in both exams and real work.

Confusing Pairs Compared

💡 Concept: Feature engineering and model selection cannot be separated. With the same data, a linear model needs scaling, binning and one-hot encoding to capture nonlinearity and scale explicitly, while a tree or boosting model needs no scaling and gains little from binning or one-hot (because trees capture nonlinearity through splits). The answer to "Which preprocessing is right?" depends on "Which model will you use?" The exam often exploits this, using the algorithm as a clue to narrow down the correct preprocessing.

Typical Paths Where Leakage Occurs

Most questions that ask you to pick the "wrong procedure" are on this list.

Week 3 Technique and Tool Map

Week 3 Wrong-Answer Notes

• Fitting the scaler or imputer on all data — the most common leakage. Fit on the training set only.
• Choosing scaling as the answer for a tree model — trees are invariant to monotonic transformations.
• Label Encoding for nominal data — it creates an order that does not exist and misleads linear models.
• One-Hot for high cardinality — dimensionality explosion. Look at target encoding, hashing and embeddings.
• Removing outliers unconditionally — in fraud and failure detection, they are the correct signal.
• Removing duplicates after the split — leakage has already happened.
• Splitting a time series randomly — the future ends up predicting the past.
• Full-period aggregate features — information from after the prediction time gets mixed in.
• A vocabulary that includes test documents — the vectorizer too must be fit on the training set only.
• Confusing Online and Offline Store — single-record low latency is Online; bulk training is Offline.

Self-Check

Answer these in your head.

1. If you simply impute missing values in right-skewed income, mean or median? → Median
2. Does XGBoost need feature scaling? → No
3. Why is one-hot unsuitable for postal codes (thousands of values)? → Dimensionality explosion
4. What makes 23:00 and 0:00 adjacent? → Sine/cosine transform
5. What happens if you fit the scaler on all data? → Data leakage
6. Which Feature Store serves real-time single-record inference lookups? → Online
7. How do you detect outliers without distribution assumptions? → IQR
8. What about extreme transaction amounts in fraud detection? → Do not remove them (they are the signal)
9. How do you handle a streaming categorical feature with fixed memory and no dictionary? → Feature Hashing
10. Which text representation reduces the influence of common words? → TF-IDF
11. When do you clean user-level duplicates? → Before the split
12. Which service keeps training and inference feature definitions consistent? → Feature Store

Question Clue → Answer Mapping

Exam Tips

• When the question names an algorithm, the preprocessing options narrow (tree → scaling is usually wrong).
• "Millions of unique values" = one-hot is wrong → think target encoding, hashing and embeddings.
• "No-code / visual / quick preparation" = Data Wrangler; "large-scale custom code" = Processing Job.
• "Training-inference consistency," "feature reuse," "single-record low-latency lookup" = Feature Store.
• Preprocessing that violates the fit on training → transform on test rule is wrong.
• An option that says "remove all outliers" is wrong in an anomaly-detection context.

Next week (Week 4) covers the second half of EDA — data visualization, distribution and correlation analysis, and dimensionality reduction (PCA), the techniques for analysis and drawing insights.

📖 Glossary

• EDA (Exploratory Data Analysis): The stage before modeling in which you examine the data's distributions, missing values, anomalies and relationships.
• MCAR / MAR / MNAR: Missing completely at random / missingness that depends on observed variables / missingness that depends on the value itself.
• Imputation: Filling in missing values with some value.
• IQR (interquartile range): Q3 − Q1. A robust criterion for setting outlier boundaries without distribution assumptions.
• Standardization / normalization: Rescaling to mean 0 and variance 1 / squeezing into the [0,1] range.
• Cardinality: The number of unique values in a categorical variable. Thousands of values means high cardinality.
• Target Encoding: Encoding that replaces a category with the target mean of that category. Watch for leakage.
• TF-IDF: A text representation that multiplies term frequency by a document-rarity weight, suppressing common words.
• Data leakage: An accident in which information that would not have been known at training time leaks into the model.
• point-in-time: The principle of using only values that were actually knowable at the prediction time.