Day 1 - Domains 1 & 2 Integration: Data Engineering + EDA

📌 Key Takeaways

• Domain 1 (20%) + Domain 2 (24%) = about 44% of the exam. This is the front part of the pipeline that "turns raw source data into clean, trainable features."
• Ingestion is the four Kinesis siblings, storage and querying is S3/Athena/Redshift, and transformation is Glue/EMR. The deciding axis is always management burden vs control.
• Encoding and scaling are decided by the model type. Distance-based and gradient-based models require scaling; tree-based models make it practically meaningless.
• Every fit (scaler, encoder, imputer, SMOTE) happens only on the training set, after the split. Breaking this rule is data leakage.
• EDA is not admiring plots but gathering the basis for the next preprocessing decision. Skewness → log transform, outliers → winsorizing, correlation above 0.95 → remove multicollinearity.

Flow Map: Until the Data Becomes Trainable

[Raw source data]
   │
   ├─ 1) Ingestion / loading (Domain 1)
   │     Real-time stream → Kinesis (Data Streams / Firehose)
   │     Batch loading    → S3 + Glue catalog
   │     Relational / migration → DMS, DataSync
   │
   ├─ 2) Storage (Domain 1)
   │     Data lake = S3 (the de facto standard)
   │     Query     = Athena (serverless), Redshift (warehouse)
   │
   ├─ 3) Transformation / ETL (Domain 1)
   │     Glue (serverless Spark) / EMR (large-scale cluster)
   │
   ├─ 4) Cleaning (Domain 2)
   │     Missing values, outliers, deduplication, type correction
   │
   ├─ 5) Feature engineering (Domain 2)
   │     Encoding, scaling, binning, dimensionality reduction (PCA)
   │
   └─ 6) Visualization / EDA (Domain 2)
         Explore distributions, correlations, outliers → QuickSight, scatter plots/histograms

Exam Weight by Domain and Typical Question Forms

Domain 1: Ingestion and Streaming (The Four Kinesis Services)

Streaming questions almost always come down to distinguishing the roles of the four Kinesis siblings.

💡 Related Theory: Streams vs Firehose is decided by "who bears the management burden." Streams manages shard capacity, consumers, and retention directly, making it the most flexible but with a heavy operational burden. Firehose has no shard concept: you only set the buffer size and time, and it delivers automatically to S3/Redshift, so it is the answer when words like "fully managed / serverless / minimal operations" appear. Answer instantly: "custom processing / milliseconds / reprocessing" is Streams, and "just drop it into S3" is Firehose.

Domain 1: Storage and ETL

Glue vs EMR is another case of "operational control vs convenience." Glue is serverless, so it provides ETL jobs and a Data Catalog (metadata) without launching a cluster, and is strong for "serverless ETL, automatic schema inference." EMR is used when you launch a cluster yourself and want fine control over the Spark version, libraries, and the Hadoop ecosystem. "Managed / serverless / quick start" = Glue; "fine-grained cluster tuning / existing Hadoop jobs" = EMR.

Domain 2: Data Cleaning

⚠️ Pitfall: Scalers, encoders, imputers, and SMOTE are fit only on the training set after the split, and only transform is applied to the validation and test sets. Fitting on the entire data leaks test information into training (data leakage), inflating only the offline score and collapsing in production. A stem like "cross-validation score is excellent but production performance is terrible" is, in most cases, a question about leakage.

Domain 2: Feature Engineering and Encoding

The distinction between "models that need" and "models that don't need" scaling falls within the exam scope. Distance-based (KNN, K-Means, SVM) and gradient-descent-based (linear/logistic regression, neural networks) models are sensitive to feature scale, so standardization is almost mandatory. Tree-based models (decision tree, random forest, XGBoost), on the other hand, have split criteria that are invariant to scale, so scaling is practically meaningless. An option like "it's XGBoost but scaling is mandatory" is a trap.

Domain 2: What Do You Look For in Visualizations

The purpose of EDA is not "a nice-looking picture" but "gathering the basis for preprocessing and modeling decisions." If the histogram shows strong skewness you decide on a log transform, if the box plot shows outliers you decide on winsorizing, and if the correlation heatmap shows two features at 0.95 or higher you decide to remove multicollinearity (drop one or use PCA). Visualization questions reduce to "what do you see in this picture and what do you do next?"

Exam Clue Keyword → Answer Translation Table (Domain 1)

Exam Clue Keyword → Answer Translation Table (Domain 2)

Confusing Pairs Comparison Tables

Streams vs Firehose vs Data Analytics

Glue vs EMR

Athena vs Redshift vs Redshift Spectrum

DMS vs DataSync

One-Hot vs Label (Ordinal) vs Target Encoding

Standardization vs Normalization

Oversampling vs Undersampling vs Class Weights

IQR vs Z-score Outlier Detection

Self-Check Questions

1. Multiple consumers must reprocess the same stream with millisecond latency? → Kinesis Data Streams
2. Automatically load a stream into S3 with no operations? → Kinesis Data Firehose
3. SQL-query an S3 data lake serverlessly? → Athena
4. Serverless Spark ETL and automatic schema inference (catalog)? → Glue
5. Is standardization mandatory before training XGBoost? → No (trees are scale-invariant)
6. Encoding for an unordered categorical (color)? → One-Hot Encoding
7. Response when two features have a correlation of 0.97? → Remove multicollinearity (drop or PCA)
8. The common transform for a strongly right-skewed distribution? → Log transform
9. What arises if you fit a scaler on the entire data? → Data leakage (fit only on the training set after the split)
10. Services for on-premises DB migration and large file transfer? → DMS / DataSync
11. To join raw S3 data from a Redshift cluster without loading? → Redshift Spectrum
12. Positives are 0.5% yet accuracy is 99%. How do you change the metric? → F1 / AUPRC (precision-recall based)
13. Outlier criterion for a heavily skewed variable? → IQR (box plot)

📖 Glossary

• Data lake: A central repository that gathers structured and unstructured data as is, without processing. On AWS this effectively means S3.
• Shard: The unit of throughput in Kinesis Data Streams. Adding shards increases throughput, and consumers read per shard.
• Data Catalog: A metadata ledger recording what data is where, with what schema. Glue crawlers fill it automatically.
• Cardinality: The number of distinct values a categorical variable can take. Thousands to millions, as with user IDs, is high cardinality.
• Data leakage: Information that should be unknowable at training time seeping in through features or preprocessing. It inflates only the offline score.
• Multicollinearity: A state where two or more features carry nearly the same information and are strongly correlated with each other. It destabilizes the coefficients of linear models.
• Skewness: The degree to which a distribution leans to one side. A long right tail is right skew, and a log transform is the common response.
• Winsorizing: Treatment that clips outliers to the upper and lower boundary values instead of discarding them. It preserves the sample count.
• AUPRC: The area under the precision-recall curve. In heavily imbalanced problems with very rare positives, it is a more trustworthy metric than accuracy or ROC-AUC.