Day 5 - Week 3 Comprehensive Review — Feature Engineering·Data Quality

📌 Key Takeaways

• The picture for Week 3 is a transform (Data Wrangler) → manage (Feature Store) → validate (Clarify) pipeline.
• The four major transformations are scaling, encoding, missing values, and outliers. Algorithm dependency (distance/gradient-based vs. tree-based) decides which one applies.
• Feature Store has two stores — online (millisecond inference) and offline (S3 training history).
• With Clarify, you look at pre-training bias (CI·DPL) and post-training explanation (SHAP).
• The one word that runs through this week is leakage — you must block transformation, synthetic, future, and evaluation leakage alike.

Feature Engineering Pipeline at a Glance

This week covered the process of refining prepared data into a form that models can learn. It is the area where the ML maxim "data over algorithm" shows most clearly.

[raw data (S3)]
        │  ① Transform (Data Wrangler)
        │     Scaling·Encoding·Missing values·Outliers
        v
[Transformed features]
        │  ② Manage (Feature Store)
        │     Online (inference) + Offline (training) stored together
        v
[Feature Group]
        │  ③ Validate (Clarify)
        │     Bias·Imbalance·Explainability analysis
        v
[Validated training dataset] → SageMaker Training

The key point is that these three stages are connected by consistency. Feature Store supplies the transformations defined in Data Wrangler identically to both training and inference, and Clarify vouches for the fairness of that data.

The statistic that data scientists spend 70–80% of their time on feature engineering is not an exaggeration. As Google's "Hidden Technical Debt in Machine Learning Systems" paper emphasizes, in real ML systems the model code is a small part, while data collection, validation, feature extraction, and consistency maintenance account for most of the complexity and the causes of failure. All three tools this week are mechanisms for taming that data-side complexity.

Four Major Feature Engineering Transformations (Day 1)

Here we review again the core transformations that turn raw data into numbers the model can read.

The deciding criterion is algorithm dependency. Distance/gradient-based algorithms (KNN, neural networks, linear regression) require scaling, but tree-based algorithms (XGBoost, Random Forest) are insensitive to scale.

from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.compose import ColumnTransformer
 
# Numeric: standardize, categorical: one-hot — bundle in one transformer
preprocessor = ColumnTransformer([
    ("num", StandardScaler(), ["age", "income"]),
    ("cat", OneHotEncoder(handle_unknown="ignore"), ["region"]),
])
X_train_t = preprocessor.fit_transform(X_train)
X_test_t = preprocessor.transform(X_test)   # test: transform only, no fit — prevents leakage

⚠️ Pitfall: Scalers and encoders are fit only on train, and only transform is applied to test. If you fit on test statistics, test information leaks into training and becomes data leakage. This principle applies equally to resampling such as SMOTE.

Data Wrangler (Day 2)

SageMaker Data Wrangler is a tool for visually importing, transforming, and analyzing data without code. It provides over 300 built-in transformations and a data quality report, and the flow you build can be exported to a Processing Job, Pipeline, or Feature Store.

The value of Data Wrangler does not end at "visual exploration" — it lies in exporting those transformations as reproducible code. Transformations hand-coded in a notebook are hard to reproduce at inference time, but a Data Wrangler flow can be exported as a Processing Job, guaranteeing the same transformation in training and inference. It is the bridge between exploration and production.

⚠️ Pitfall: The data you see on screen is a sample. To apply the flow to the full dataset, you must export the flow as a Processing Job and run it.

Feature Store (Day 3)

SageMaker Feature Store is a repository for centrally storing, reusing, and consistently serving features. Two stores are the core.

The greatest value is preventing training/serving skew. Training and inference share the same feature definition, which eliminates at the root the problem where the two compute things slightly differently and performance collapses. The Feature Group (Record Identifier + Event Time) is the unit, and Event Time is the basis for point-in-time queries and for preventing future leakage.

# Training: bulk query of the offline store via Athena
query = feature_group.athena_query()
query.run(query_string="SELECT * FROM customer_features WHERE ...",
          output_location="s3://my-bucket/query-results/")
 
# Inference: fetch the latest single record from the online store
record = featurestore_runtime.get_record(
    FeatureGroupName="customer-features",
    RecordIdentifierValueAsString="cust_12345",
)

Clarify and Data Quality (Day 4)

SageMaker Clarify measures bias before and after training (sensitive attribute facet, CI·DPL metrics) and explains the feature contributions to a prediction with SHAP. Adding data quality checks on top completes a trustworthy dataset.

The one word that runs through this week is "leakage". Fitting on test is transformation leakage, resampling before the split is synthetic leakage, using future values as features is future leakage, and ignoring Feature Store's Event Time is look-ahead leakage. Every leakage produces the same symptom: "evaluation looks good, but deployment fails". Half of feature engineering and data quality is preventing leakage.

Leakage Types Summarized

Week 3 One-Line Checklist

Reviewing Week 3 with Scenarios

Linking Transform, Manage, and Validate Together

[raw]
  │  Data Wrangler flow (design)
  │      └─ export → Processing Job (apply to full data)
  v
[features]
  │  Feature Store ingest
  │      ├─ offline → training set (Athena, point-in-time)
  │      └─ online  → inference (GetRecord)
  v
[validation]
  │  Clarify: pre-training bias (CI·DPL) / post-training SHAP
  │  Quality: imbalance handling (after split), stratified·temporal split, test once
  v
[Trustworthy training dataset]

If any single arrow in this picture is broken by hand-coding, reproducibility and consistency collapse together. "Preprocessing that only works in a notebook" will inevitably cause problems at deployment time. On the exam, when you see phrases such as "reproducibly" or "identical in training and inference", the answer points toward export, Feature Store, and Pipelines.

Week 3 in One Sentence

"Fit only on train, split first of all, define features in one place so training and inference read from it together, and measure bias before you build."

Check Before Moving to Week 4

Next week (Week 4) we move into full-scale model training, tuning, and evaluation with the data prepared this way.

📖 Glossary

• Feature engineering: The whole task of converting raw data into numeric representations a model can learn from.
• ColumnTransformer: A pipeline tool that applies different transformations to numeric and categorical columns at once.
• handle_unknown="ignore": An encoder option that handles a new category not seen during training as zeros without raising an error.
• flow: A blueprint in Data Wrangler that fixes the order of transformations. The key to reproducibility.
• Feature Group: The storage unit of Feature Store. Record Identifier and Event Time are required.
• Event Time: The time at which a feature value was recorded. The basis for reconstructing past points in time and preventing future leakage.
• training/serving skew: The phenomenon where feature computation differs between training and inference, so the model silently gets worse.
• facet (sensitive attribute): A column whose fairness must be examined. The reference axis for Clarify bias analysis.
• CI / DPL: Bias metrics measuring the imbalance in sample counts between groups / the difference in positive-label proportions between groups.
• SHAP: An explanation metric computed by fairly distributing each feature's share of contribution to a prediction.