Day 2 - Feature Selection: Filter, Wrapper and Embedded Methods, Importance, and Multicollinearity

📌 Key Takeaways

• Feature selection keeps a subset of the original features, so interpretability is preserved. PCA creates new axes and loses meaning.
• Three approaches: filter (fast, model-agnostic) / wrapper (accurate, very slow) / embedded (balanced; Lasso and tree importance).
• L1 (Lasso) drives coefficients to exactly 0, giving automatic selection. L2 (Ridge) only shrinks them.
• For importance, impurity-based (MDI) importance is biased toward high-cardinality features. Permutation importance and SHAP are more trustworthy.
• Multicollinearity is diagnosed with the correlation matrix (|r| > 0.8–0.9) and VIF (> 5 caution, > 10 severe). It is fatal to linear models; trees are insensitive.

Why Feature Selection?

If Day 1's dimensionality reduction was "creating new synthetic axes," feature selection is "keeping only the useful existing features." Both reduce dimensions, but selection preserves the interpretability of the original features.

• Less overfitting: irrelevant features let the model learn noise
• Faster training and inference
• Higher interpretability: the fewer the features, the easier the model is to explain
• Mitigates the curse of dimensionality

Feature selection and PCA both reduce dimensions, but their philosophies differ. PCA linearly combines all the original features to create new axes, so the physical meaning of the result is lost. Selection keeps a subset of the original features as they are, so it is preferred in regulated industries (finance, healthcare) where you must explain "which variables were used for prediction."

Three Approaches

Which Approach Should You Pick?

Filter Methods

Each feature is evaluated independently by its relationship to the target. There is no model training, so this is the fastest approach.

from sklearn.feature_selection import SelectKBest, f_classif, mutual_info_classif
 
# Select the top k features with the ANOVA F-test
selector = SelectKBest(score_func=f_classif, k=20)
X_filtered = selector.fit_transform(X_train, y_train)
 
# Use mutual information for nonlinear relationships
mi_selector = SelectKBest(score_func=mutual_info_classif, k=20)

Which statistic to use in a filter is determined by the combination of feature type and target type.

Wrapper Method: RFE

RFE (Recursive Feature Elimination) repeats the process of training a model and removing the weakest feature.

from sklearn.feature_selection import RFE
from sklearn.ensemble import RandomForestClassifier
 
rfe = RFE(estimator=RandomForestClassifier(), n_features_to_select=15)
rfe.fit(X_train, y_train)
selected = X_train.columns[rfe.support_]

Embedded Method: L1 Regularization

from sklearn.linear_model import Lasso
 
lasso = Lasso(alpha=0.01)
lasso.fit(X_scaled, y_train)
# Features with a coefficient of 0 are removed automatically
selected = X_train.columns[lasso.coef_ != 0]

      L1 (Lasso) constraint     L2 (Ridge) constraint
          ╱╲                        ╭───╮
        ╱    ╲                     │     │
        ╲    ╱                     │     │
          ╲╱                        ╰───╯
   Loss contours easily meet     The contact point rarely lands on an axis
   at a vertex → coefficient = 0 → it only gets close to 0, never exactly 0
        (sparse solution)            (shrinkage only)

Lasso performs feature selection because of the geometry of the L1 penalty. The L1 constraint region is diamond-shaped, so the loss contours are likely to meet at a vertex (on an axis, i.e., coefficient = 0). L2 (Ridge) is circular, so although it shrinks coefficients toward 0, it rarely sets them exactly to 0. That is why "if you need a sparse solution, use L1."

⚠️ Pitfall: Lasso penalizes coefficient magnitude, so scaling must come first. Without standardization, the coefficient of a large-unit variable looks small and gets pushed to 0 first.

⚠️ Pitfall: Feature selection is also a data-dependent transformation. If you compute correlations or importance on all the data, select features, and then cross-validate, validation-set information leaks into the selection. Selection must be put in the pipeline and performed only on the training portion of each fold.

Feature Importance

Tree-based models provide feature importance after training.

import shap
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_test)
shap.summary_plot(shap_values, X_test)   # Visualize feature contributions

Impurity-based importance is biased toward overestimating features with many split candidates (continuous, many-category). Permutation importance randomly shuffles a feature in a trained model and measures the performance drop, so it avoids this bias. SHAP additively decomposes feature contributions for each prediction, providing both global and local explanations, but is expensive to compute. For the exam, remember: "impurity-based is biased toward high cardinality; permutation and SHAP are more trustworthy."

Multicollinearity

When two or more features are strongly correlated, multicollinearity arises. In linear models, coefficients become unstable and interpretation is distorted.

Diagnostic metrics:

from statsmodels.stats.outliers_influence import variance_inflation_factor
import pandas as pd
 
vif = pd.DataFrame()
vif["feature"] = X.columns
vif["VIF"] = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]

Remedies: remove one of the correlated features, merge them with PCA, or mitigate with L2 regularization (Ridge).

Multicollinearity Diagnosis → Remedy

Multicollinearity Sensitivity by Model

VIF is computed as VIF = 1/(1−R²), where R² is the coefficient of determination from regressing one feature on all the other features. As R² approaches 1 (that is, as the feature becomes almost perfectly predictable from the others), VIF explodes. Note: tree-based models are insensitive to multicollinearity — because each split uses only one feature at a time. Multicollinearity is fatal in linear models, where coefficient interpretation matters.

When two correlated features exist in a tree model, the importance is split between them and each shows up low. If you judge "the importance is low, so it can be discarded," you may in fact throw away an important signal wholesale.

Practical Selection Order

Feature selection is not a one-shot job but a funnel that filters in order from the cheapest step.

[Hundreds to thousands of raw features]
        │
  ① Remove constant · quasi-constant columns   (variance threshold. Axes with no information)
        │
  ② Remove ID · suspected-leakage columns      (columns that make the model memorize rows)
        │
  ③ Filter: score by correlation · F-test · mutual information → top k
        │
  ④ Clean up multicollinearity                 (|r| > 0.8–0.9, VIF > 5–10)
        │
  ⑤ Embedded: further reduction with Lasso or tree importance
        │
  ⑥ (If you can afford it) Wrapper: final fine-tuning with RFE
        │
  ⑦ Confirm performance with cross-validation — do ③–⑥ only on the training portion of each fold

AWS Service Mapping

Question Clue → Answer Mapping

The next post covers data visualization, for checking distributions and correlations by eye, and AWS's BI service, QuickSight.

📖 Glossary

• Feature selection: Dimensionality reduction that picks and keeps only the useful features among the original ones. Column names are preserved.
• Filter method: A way of scoring and picking features using only statistics such as correlation coefficients or chi-square, without training a model.
• Wrapper method: A way of finding the best combination by training and evaluating an actual model while changing feature combinations. Accurate but slow.
• Embedded method: A way in which selection happens together with the model's training (Lasso, tree importance).
• RFE (Recursive Feature Elimination): A wrapper technique that repeats training a model, dropping the weakest feature, and training again.
• Mutual information: A measure of how much information two variables share. Also captures nonlinear relationships.
• Sparse solution: A solution in which many coefficients are exactly 0. Produced by L1 regularization.
• Impurity-based importance (MDI): Importance obtained by summing the impurity reduction when a tree splits. Overestimates features with many split candidates.
• Permutation importance: Importance measured by how much performance drops after the feature's values are randomly shuffled.
• Multicollinearity / VIF: A state in which features are strongly correlated with each other / a metric measuring that degree. Above 5 is caution, above 10 is severe.