Feature store patterns, training/serving skew prevention, feature pipelines for ML teams, point-in-time correct joins, and bridging data engineering with MLOps conventions. Use this skill whenever an ML team needs feature pipelines, when building a feature store or deciding whether to use one, when there's a training/serving skew problem (model performance in production differs from validation), when features need to be shared across multiple models, or when designing point-in-time correct feature computation. Also trigger when the user mentions feature stores (Feast, Tecton, Hopsworks), label leakage, backfilling features, offline/online store separation, or when data engineering work feeds directly into model training. If the words "features", "training set", "model pipeline", or "MLOps" appear alongside data engineering questions, this skill should be active.
Feature store patterns, training/serving skew prevention, feature pipelines for ML teams, point-in-time correct joins, and bridging data engineering with MLOps conventions. Use this skill whenever an ML team needs feature pipelines, when building a feature store or deciding whether to use one, when there's a training/serving skew problem (model performance in production differs from validation), when features need to be shared across multiple models, or when designing point-in-time correct feature computation. Also trigger when the user mentions feature stores (Feast, Tecton, Hopsworks), label leakage, backfilling features, offline/online store separation, or when data engineering work feeds directly into model training. If the words "features", "training set", "model pipeline", or "MLOps" appear alongside data engineering questions, this skill should be active.
ML Feature Engineering Patterns
The Data Engineering / ML Boundary
Data engineers own the data. ML engineers own the models. Feature engineering sits at the boundary — and when it's designed poorly, both sides pay for it. The most common failures:
Training/serving skew — features computed differently at training time vs. prediction time → model performs worse in production than in validation
Label leakage — features computed using data from the future, making the training set artificially easy → model fails in production
Feature duplication — each ML team recomputes the same features independently → inconsistent definitions, wasted compute
The patterns in this skill address all three.
Point-in-Time Correct Joins
This is the most important concept in ML feature engineering. A model trained to predict churn should only "see" data that would have been available at the time of the prediction — not data from the future.
The wrong way
-- BAD: This leaks future data into training
u.user_id,
u.subscription_tier,
(e.event_id) events_last_30d,
u.churned label
users u
events e u.user_id e.user_id
e.event_date DATEADD(, , )
u.label_date
, , ;
-- If we're predicting churn as of 2024-01-15, we shouldn't know about
-- events that happened on 2024-01-20
SELECT
COUNT
AS
-- counts events AFTER the label date!
AS
FROM
JOIN
ON
=
AND
>=
day
-30
CURRENT_DATE
-- wrong: uses today's date
WHERE
=
'2024-01-15'
GROUP
BY
1
2
4
The right way — as-of join
-- GOOD: Features are computed as of the label date, using only past dataSELECT
u.user_id,
u.subscription_tier,
COUNT(e.event_id) AS events_last_30d,
u.churned AS label
FROM user_labels u -- contains (user_id, label_date, churned)LEFTJOIN events e ON u.user_id = e.user_id
AND e.event_date >= DATEADD(day, -30, u.label_date) -- relative to label dateAND e.event_date < u.label_date -- only past eventsGROUPBY1, 2, 4;
The rule: every date range in a feature join must be bounded by < label_date on the upper end.
Point-in-time with slowly-changing dimensions
When a dimension table changes over time (e.g., a user's subscription tier changes), you need the value that was true at label_date, not the current value.
-- Get the subscription tier each user had at each label date-- (assuming subscription_changes has valid_from/valid_to columns)SELECT
l.user_id,
l.label_date,
s.subscription_tier AS tier_at_label_date
FROM user_labels l
JOIN subscription_changes s ON l.user_id = s.user_id
AND s.valid_from <= l.label_date
AND (s.valid_to > l.label_date OR s.valid_to ISNULL)
Feature Pipeline Architecture
A feature pipeline has two paths:
Offline — batch computation, used for training and bulk scoring
Online — low-latency lookup, used for real-time predictions
Offline pipeline:
Raw events → Transform → Feature store (offline) → Training dataset → Model training
Online pipeline:
Raw events → Transform → Feature store (online) → Real-time prediction API
The transform logic must be identical in both paths. If it isn't, you get training/serving skew.
The serving skew trap
# Training time (Python/Pandas)
df["days_since_login"] = (df["label_date"] - df["last_login_date"]).dt.days
# Serving time (different code, different team)
days_since_login = (datetime.now() - user.last_login).days # subtly different
Any divergence in feature computation between training and serving will degrade model performance in production. The model won't fail — it will just quietly perform worse.
Prevention: Keep feature logic in a single, shared function called by both the training pipeline and the serving code. Write integration tests that verify identical output for both paths on the same input.
# Shared feature computation — called by BOTH training pipeline and servingdefcompute_engagement_features(user_id: str, as_of_date: date) -> dict:
events = db.query("""
SELECT COUNT(*) AS event_count,
MAX(event_date) AS last_event_date
FROM events
WHERE user_id = %s AND event_date < %s
AND event_date >= %s
""", user_id, as_of_date, as_of_date - timedelta(days=30))
return {
"events_last_30d": events["event_count"],
"days_since_last_event": (as_of_date - events["last_event_date"]).days
if events["last_event_date"] elseNone
}
Feature Store Patterns
A feature store is a system that manages feature computation, storage, and serving. Whether you use a managed one (Feast, Tecton, Hopsworks) or a simple internal one, the key components are the same.
When you need a feature store
Multiple models need the same features (churn model, LTV model, propensity model all need engagement features)
Features need to be available both for offline training and online serving
You want a catalog of what features exist and who owns them
You need to backfill features for historical label dates
If only one model exists and it only runs in batch, you may not need a full feature store — a well-structured dbt model is sufficient.
Simple internal feature store (dbt + BigQuery)
-- The feature table — one row per (entity, feature_date)-- Materialized as an incremental dbt model-- models/feature_store/user_engagement_features.sql
{{
config(
materialized='incremental',
unique_key=['user_id', 'feature_date'],
partition_by={'field': 'feature_date', 'data_type': 'date'},
)
}}
SELECT
user_id,
DATE(event_date) AS feature_date,
COUNT(*) AS events_last_30d,
COUNT(DISTINCT session_id) AS sessions_last_30d,
MAX(event_date) AS last_event_date,
DATE_DIFF(DATE(event_date), MAX(event_date), DAY) AS days_since_last_event
FROM {{ ref('stg_events') }}
{% if is_incremental() %}
WHERE event_date >= DATE_SUB((SELECTMAX(feature_date) FROM {{ this }}), INTERVAL31DAY)
{% endif %}
GROUPBY1, 2
Training time: query the feature table with a point-in-time join against your label table.
Serving time: query the feature table for feature_date = today().
Feast (managed feature store)
from feast import FeatureStore, Entity, Feature, FeatureView, FileSource
from feast.types import Float64, Int64
# Define entity
user = Entity(name="user_id", join_keys=["user_id"])
# Define feature view
user_engagement_fv = FeatureView(
name="user_engagement",
entities=[user],
ttl=timedelta(days=1),
features=[
Feature(name="events_last_30d", dtype=Int64),
Feature(name="days_since_last_event", dtype=Int64),
],
source=BigQuerySource(table="project.dataset.user_engagement_features"),
)
store = FeatureStore(repo_path=".")
# Training — get historical features (point-in-time correct automatically)
training_df = store.get_historical_features(
entity_df=label_df, # contains user_id and event_timestamp (the label date)
features=["user_engagement:events_last_30d", "user_engagement:days_since_last_event"],
).to_df()
# Serving — get online features (millisecond latency from Redis/DynamoDB)
features = store.get_online_features(
features=["user_engagement:events_last_30d"],
entity_rows=[{"user_id": "user_123"}],
).to_dict()
Label Design
Churn label example
-- Define churn: no activity for 30 days after a 90-day observation window-- Label date = the date we're making the prediction-- We look 30 days forward from label_date to determine if they churnedSELECT
user_id,
label_date,
CASEWHENMAX(event_date) < DATEADD(day, 30, label_date) THEN1ELSE0ENDAS churned
FROM (
-- Generate label dates (one per user per week)SELECTDISTINCT user_id, date_spine.date AS label_date
FROM active_users
CROSSJOIN date_spine
WHERE date_spine.date BETWEEN'2023-01-01'AND'2023-12-31'
) label_candidates
LEFTJOIN events e ON label_candidates.user_id = e.user_id
AND e.event_date BETWEEN label_date AND DATEADD(day, 30, label_date)
GROUPBY1, 2
Important: label generation requires future data. The label for a given label_date can only be computed 30 days later. Don't use recent dates as labels in training — the churned flag won't be set yet.
Feature Documentation for ML Teams
When data engineers hand features to ML teams, document the grain, computation window, and known caveats:
# feature_catalog/user_engagement.yamlfeature_group:user_engagemententity:user_idgrain:"One row per (user_id, feature_date)"update_frequency:dailyowner:data-engineering@company.comfeatures:-name:events_last_30dtype:integerdescription:"Count of all events in the 30 days prior to feature_date"computation_window:"feature_date - 30 days to feature_date (exclusive)"known_caveats:"Null for users who signed up < 30 days before feature_date"-name:days_since_last_eventtype:integerdescription:"Days between feature_date and the most recent event before feature_date"computation_window:"All history up to feature_date"known_caveats:"Null for users with no recorded events"
Checklist: Before Handing Features to the ML Team
Point-in-time correctness verified — all feature windows are bounded by < label_date
Feature computation function is shared between training pipeline and serving code
Null values documented — what does a null mean for each feature?
Feature distribution checked after backfill (spot-check means and percentiles per month)
Label leakage audit — no feature references data that would only be known after the label event
Features versioned — if computation logic changes, old features remain accessible
Feature documentation written (grain, window, caveats, owner)
Integration test confirms offline and online paths produce identical output for same input
