Day 3 - SageMaker Feature Store: Managing Features as Assets

📌 Key Takeaways

• Feature Store is a repository that centrally stores, reuses, and consistently serves features. Its unit is the Feature Group.
• The required elements of a Feature Group are the Record Identifier (row identifier) and the Event Time (time of recording).
• The online store = millisecond single-record lookups (real-time inference), the offline store = S3 Parquet history (training and batch).
• Its greatest value is preventing training/serving skew — training and inference share the same feature definition.
• A point-in-time correct join based on Event Time prevents future information leakage (look-ahead bias).

What Is Feature Store

If the features you built yesterday exist only in a dataframe in a notebook, the next project has to build the same features again. Worse, the transformation used at training time and the one used at inference time can differ subtly, so model performance quietly collapses.

Feature Store is a repository that centrally stores, manages, and serves ML features. Its core unit is the Feature Group, which is made of rows and columns like a relational table, with each column being one feature.

from sagemaker.feature_store.feature_group import FeatureGroup
 
feature_group = FeatureGroup(name='customer-features', ...)
 
feature_group.create(
    record_identifier_name='customer_id',   # Row identifier
    event_time_feature_name='event_time',    # Event time
    role_arn=role,
    enable_online_store=True                 # Enable the online store
)
 
# Ingest features
feature_group.ingest(data_frame=df, max_workers=3)

When multiple records with different event_time values accumulate for the same customer_id, you can reconstruct "what was the feature value at a particular point in time."

History of customer_id = cust_123

event_time          purchase_30d   activity_score
2026-01-15 10:00        12             0.71
2026-03-02 09:30        18             0.83
2026-06-24 14:10        25             0.66   ← the "latest value" the online store returns

The offline store keeps all three rows above → past points in time can be reconstructed

Event Time is related to the bitemporal concept in time series databases. When training a model on past data, you must use only "the feature values that could actually be known at that time" to prevent future information leakage (look-ahead bias). For example, if you predict a January 2024 transaction using a feature value that was updated in March 2024, you have seen the future. Event Time-based lookup is the mechanism that guarantees this temporal consistency.

Online Store and Offline Store

The core of Feature Store is that it provides two kinds of stores at the same time. On the exam you need to distinguish the two.

# Online store: look up the latest features during real-time inference
record = featurestore_runtime.get_record(
    FeatureGroupName='customer-features',
    RecordIdentifierValueAsString='cust_12345'
)
 
# Offline store: build a training dataset (Athena)
query = feature_group.athena_query()
query.run(query_string="SELECT * FROM customer_features WHERE ...",
          output_location='s3://my-bucket/query-results/')

⚠️ Trap: If you enable only the online store, training history is not accumulated in S3, and if you enable only the offline store, you cannot do low-latency lookups in real-time inference. Most real ML systems enable both, so the same features are used consistently for training (offline) and inference (online). When you ingest data, SageMaker automatically syncs the latest value to the online store and the history to the offline store.

Consistency: Preventing training/serving Skew

The greatest value of Feature Store is that training and inference share the same feature definition. Training/serving skew is the classic problem in which the way features are computed differs between training and serving, degrading model performance.

A typical scenario: the training pipeline computes "average purchase amount over the last 30 days" with pandas, and the inference server recomputes the same value in Java. If the two implementations differ subtly (boundary handling, rounding), the values the model sees differ from those at training time.

[Feature computation pipeline]  ──ingest──>  [Feature Store]
                                                |
                          ┌─────────────────────┴──────────────────┐
                          v                                          v
                [Offline store]                              [Online store]
                Build training datasets                  Real-time inference lookups
                (Athena, same definition)                (GetRecord, same definition)

Training/serving skew is a core problem emphasized in Google's paper "Hidden Technical Debt in Machine Learning Systems," which dealt with the technical debt of ML systems. Model code is a small part of the whole ML system, and data pipelines and feature consistency are the main causes of failure. Feature Store reduces this debt by promoting features from "code" to a "shared data asset," gathering the definition in one place.

Feature Reuse and Governance

The second value of Feature Store is reuse. Features such as "customer lifetime value" and "recent activity score" built by Team A can be taken as they are by Team B for a new model.

Feature Store has a capability to join several Feature Groups with point-in-time consistency. When building training data, if you join customer features and product features on their respective event times, only "the values that could be known at that time" are combined exactly. This is called a point-in-time correct join, and it automates the prevention of future leakage, which is very tricky to do by hand.

Wrong join (future leakage)              Correct point-in-time join
Prediction time: 2026-01-15              Prediction time: 2026-01-15
customer feature: value of 2026-03-02 ✗  customer feature: latest value before 2026-01-15 ✓
  → Saw information from two months later    → Uses only values known at that time

A fintech company ran two models, fraud detection and credit scoring, and both needed a feature called "recent transaction frequency." At first each team computed it separately, so the definitions differed subtly, and when one team fixed a bug it was not reflected in the other. After this feature was defined once and shared through Feature Store, the definition mismatch disappeared, and fixing one place improved both models together.

Data Flow Summary

A typical pipeline connects like this.

This whole flow shares the same feature definition, so consistency is maintained.

Exam discrimination summary

Store Enablement Combinations and Results

When you create a Feature Group, which of online and offline you turn on determines what you can do afterward.

• With both turned on, a single ingest reflects the latest value in the online store and the history in the offline store.
• If you later need training history but had the offline store turned off, the history for that period cannot be recovered.

Which Features to Put in Feature Store

Putting in every column is not the answer. There is a balance between management cost and value.

The Moment Point-in-Time Consistency Breaks

Time to predict: 2026-01-15
   │
   ├─ Correct: use the latest record among those with event_time <= 2026-01-15
   │
   └─ Wrong: use the latest record (2026-06-24) as it is
                → Effectively feeds information from 5 months later into training

⚠️ Trap: When backtest performance is unusually good, doubt point-in-time consistency first, before celebrating the performance. A training set that did not use a point-in-time correct join almost always produces optimistic numbers.

In the next lesson we look at data bias and quality, and SageMaker Clarify, which check whether the data prepared this way is fair and balanced.

📖 Glossary

• Feature Store : A repository that gathers ML features centrally and serves them consistently to training and inference.
• Feature Group : The basic unit of Feature Store. A table of rows and columns where the features become the columns.
• Record Identifier : The key that distinguishes each row. For online lookups, the latest record is found with this value.
• Event Time : The time at which that feature value was recorded. The basis for reconstructing past points in time and preventing leakage.
• Online store : A key-value store that returns 1 latest feature record within milliseconds. For real-time inference.
• Offline store : A store that accumulates all history in S3 as Parquet. For building training datasets.
• GetRecord : The API that fetches one latest record by identifier from the online store.
• training/serving skew : The phenomenon where feature computation differs between training and inference, so the model quietly gets worse.
• point-in-time correct join : A join method that combines only "the values that could be known at that time" relative to the prediction time.
• look-ahead bias (future leakage) : An error in which information after the prediction time is used in training, inflating performance.