Day 3 - Time Series, Text Features, and High-Cardinality Categorical Handling

📌 Key Takeaways

• Don't feed a date in as-is; decompose it into components (year/month/day/day of week/hour/weekend-holiday flags, elapsed days).
• Cyclical variables (hour, day of week, month) must be mapped onto the unit circle with two axes, sine and cosine, so that 23:00 and 00:00 become neighbors.
• Text vectorization gains expressive power in the order BoW → TF-IDF → embeddings. TF-IDF suppresses common words.
• One-hot on a high-cardinality categorical explodes dimensionality. Control it with target/frequency encoding, hashing, or embeddings.
• Hashing needs no vocabulary, so it is strong on unseen categories and streaming, but you accept collisions.

Derived Features from Dates and Times

Yesterday we covered numeric and general categorical data. Today we look at feature engineering for two specialized data types that need separate handling — dates and times (time series) and text — and at the perennial encoding problem of high-cardinality categorical variables. MLS-C01 often asks about this topic in the form "which transformation is appropriate for which data?"

Using a date column as-is (2026-06-26 14:30:00) leaves the model unable to read any meaning. A date must be decomposed into components.

import pandas as pd
 
df["ts"] = pd.to_datetime(df["timestamp"])
df["month"] = df["ts"].dt.month
df["dayofweek"] = df["ts"].dt.dayofweek
df["hour"] = df["ts"].dt.hour
df["is_weekend"] = (df["dayofweek"] >= 5).astype(int)

Cyclical variables hide a trap. Hour 23 and hour 0 are really 1 hour apart, but left as integers they look 23 apart. Sine/cosine transformation restores the circular structure.

import numpy as np
 
df["hour_sin"] = np.sin(2 * np.pi * df["hour"] / 24)
df["hour_cos"] = np.cos(2 * np.pi * df["hour"] / 24)

[Integer encoding]   0 ─ 1 ─ 2 ─ ... ─ 22 ─ 23        distance between 23 and 0 = 23 (wrong)

[sin/cos]            0:00
                  23 ╱ ╲ 1              23:00 and 00:00 are neighbors on the unit circle
                    │   │
                  18 ╲ ╱ 6              distance = short (correct)
                     12:00

The essence of sine/cosine encoding is to "preserve periodic distance." When you map hours as angles on the unit circle, 23:00 (345°) and 00:00 (0°) become adjacent on the circle, so the model treats them as close. Using sine alone creates ambiguity (12:00 and 00:00 get the same value), so sine and cosine must be used together for all 24 hours to have unique (x, y) coordinates. This technique applies to every cyclical variable, such as day of week and month.

Features Specific to Time Series

• Lag features: Use past values directly as features, such as yesterday's sales or sales 7 days ago.
• Rolling statistics: Window statistics such as the 7-day moving average and moving standard deviation.
• Difference: The change from the previous day. Removes the trend to achieve stationarity.
• Seasonal indicators: Week of the year, quarter, promotion-period flags.

⚠️ Pitfall: Lag and rolling features are instant leakage if data after the prediction time gets mixed in. If a "last 7 days average" includes the prediction day itself or later values, you have trained on information that would not have been known in actual operation. Time series must also always be split in chronological order.

Text Features: BoW, TF-IDF, Embeddings

Converting text into numeric vectors is called vectorization. Let's go through approaches from simplest to most sophisticated.

TF-IDF fixes BoW's fundamental weakness. Words like "the" and "is" appear in every document and have no discriminative power, but BoW only counts frequency and so overweights them. TF-IDF lowers the weight of words that appear in many documents, bringing out rare but meaningful words.

from sklearn.feature_extraction.text import TfidfVectorizer
 
vectorizer = TfidfVectorizer(
    max_features=5000,      # top 5000 words only
    ngram_range=(1, 2),     # unigrams + bigrams
    stop_words="english",   # remove stopwords
)
X_text = vectorizer.fit_transform(train_texts)
X_test_text = vectorizer.transform(test_texts)   # transform only

TF-IDF's IDF (inverse document frequency) = log(total number of documents / number of documents containing that word). A word that appears in every document has a ratio near 1, so its log approaches 0 and its weight vanishes, while a word appearing in only a few documents gets a large IDF. It implements the information-theoretic principle "ignore the common, highlight the rare" by multiplication. However, TF-IDF treats words independently, so it misses semantic similarity such as between "good" and "great." Embeddings overcome this by placing words in a dense vector space.

On AWS, instead of vectorizing text yourself, you can use Amazon Comprehend (entity, sentiment, and key-phrase extraction) or create embeddings with SageMaker's built-in BlazingText (a Word2Vec implementation).

Other Things That Come Up in Text Preprocessing

• Tokenization: Cutting sentences into word or subword units.
• Stopword removal: Dropping function words such as particles in Korean or "the/is" in English.
• Lemmatization and stemming: Grouping "running/ran/runs" into one.
• Lowercasing and special-character cleanup: Prevents the same word from scattering into several forms.

⚠️ Pitfall: The vectorizer's vocabulary must also be fit on the training set only. Building the vocabulary with the test documents included is leakage. It is normal behavior for words seen for the first time in the test set to simply be ignored.

High-Cardinality Categorical Handling

Categoricals with thousands to millions of unique values, such as cities, product IDs, and user IDs, explode in dimensionality if one-hot encoded. Here is a summary by strategy.

The hashing trick maps categories into a fixed-size vector with a hash function. No vocabulary has to be built in advance, so it is strong on unseen categories and streaming data. The price is collisions — different categories may land in the same slot.

from sklearn.feature_extraction import FeatureHasher
 
hasher = FeatureHasher(n_features=256, input_type="string")
X_hashed = hasher.transform(df["city"].astype(str).apply(lambda x: [x]))

[One-hot]    5000 cities → 5000 columns (sparse, memory explosion)
[Hashing]    5000 cities → fixed 256 columns (accept collisions)
[Target]     5000 cities → 1 column (watch for leakage)
[Embedding]  5000 cities → 16-dimensional dense vector (obtained by learning)

An embedding learns a mapping from a high-dimensional sparse (one-hot) representation to a low-dimensional dense one. Each category is mapped to, say, a 16-dimensional real vector, and this vector is optimized together during model training. As a result, "categories that behave similarly end up close in vector space" (e.g., in a recommender system, users with similar tastes become adjacent). It is the same idea as word embeddings and is the standard in deep-learning recommendation and NLP. As a rule of thumb, the embedding dimension is often set to about the fourth root of the cardinality.

Blindly applying one-hot to a high-cardinality categorical blows up memory and, because of sparsity, makes training difficult too. When you see a description like "millions of unique values," one-hot is the wrong answer, and you should think of target encoding, hashing, and embeddings.

Before Feeding a Time Series into a Model

Apart from derived date features, there are concepts worth knowing when handling the time series itself.

• Trend: A component that rises or falls over the long term.
• Seasonality: A component that repeats on a daily, weekly, monthly, or yearly basis.
• Stationarity: The property that mean and variance do not change over time. Many traditional time-series models assume it.
• Differencing: A transformation that takes the difference from the previous value to remove the trend and move closer to stationarity.

The moment you shuffle time-series data randomly to split train/validation, you are predicting the past with the future. For time series, a chronological split is the rule, and cross-validation likewise slides the window from front to back.

Example Text Processing Pipeline

Text has a long chain of transformation steps, so it is easy to get the order wrong. The typical order is as follows.

Raw text ─▶ Normalization (lowercasing, special-character cleanup)
         ─▶ Tokenization
         ─▶ Stopword removal
         ─▶ Lemmatization/stemming
         ─▶ Vectorization (TF-IDF or embeddings)   ← fit on the training set only
         ─▶ Dimensionality reduction (optional, SVD, etc.)
         ─▶ Model input

• In tasks where negation matters, such as sentiment analysis, remove the negation words from the stopword list.
• A TF-IDF result is a very sparse high-dimensional matrix, so add dimensionality reduction such as SVD if needed.
• If you use embeddings, much of this preprocessing is absorbed into the model.

Transformation Summary by Data Type

In the next note we look at the SageMaker tools that automate these transformations at scale — Data Wrangler, Processing Jobs, and Feature Store.

📖 Glossary

• Derived feature: A new feature computed from an original column. Date decomposition is the classic example.
• Cyclical variable: A variable whose end connects to its start, such as hour, day of week, or month. Needs sin/cos transformation.
• Lag feature: A value from a past time point brought into the current row as a feature.
• Vectorization: Converting non-numeric data such as text into numeric vectors.
• Bag of Words: The simplest text representation, which ignores word order and counts only occurrence frequency.
• TF-IDF: A representation that multiplies term frequency by a document-rarity weight to suppress common words.
• Embedding: A representation of a category or word as a low-dimensional dense vector. Obtained by learning.
• Feature Hashing (hashing trick): An encoding that maps categories to fixed dimensions with a hash function. No vocabulary needed.
• Collision: The phenomenon of different categories being assigned to the same hash slot. The price of hashing.
• IDF (inverse document frequency): A weight that grows as fewer documents contain the word. It dampens common words.