ワンクリックで
feature-store-validator
Validates feature store configurations, feature definitions, freshness, lineage, and online/offline consistency.
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
メニュー
Validates feature store configurations, feature definitions, freshness, lineage, and online/offline consistency.
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
SOC 職業分類に基づく
Reference copy of the operating contract for the defense lens at a human gate — input binding, the six sources of the case for crossing, the untrusted-input and exhibit-marker rules, the evidence whitelist, the degraded-input table, the severity/confidence/variance scales, the output contract, and the escalation table. NOT preloaded into any agent. agents/iron-loop/advocate-critic.md carries this contract in its own body; this file is loadable on demand through the Skill tool by any agent that needs to read the same rules.
Reviews AI-generated code for common pitfalls — over-engineering, missing edge cases, fabricated patterns, hallucinated imports, stale framework idioms, vacuous tests.
Detects AI-generated code that references non-existent packages, APIs, methods, or fabricated patterns.
Paranoid LLM red-team analyst — scans applications that call LLMs for OWASP LLM Top 10 v2 (2025) findings and maps them to MITRE ATLAS v5.4.0 adversary tactics.
Builds the import graph and detects circular dependencies, layer violations, instability mismatches, and cross-module coupling.
Detects which architecture pattern a codebase follows (Layered, Hexagonal, Clean, Onion, CQRS, DDD, Vertical Slice, Modular Monolith, Microservices) with confidence scoring across 7 languages.
| name | feature-store-validator |
| description | Validates feature store configurations, feature definitions, freshness, lineage, and online/offline consistency. |
| type | skill |
| when_to_load | ["feature store check","feature consistency","feature drift","feature store validation","feature store audit","feature lineage"] |
| related_skills | ["data-ml/data-quality-checker","data-ml/ml-model-validator","specialized/database-reviewer"] |
| effort_level | high |
| tools | Bash, Read |
| model | opus |
| tier | 2 |
| dispatch_protocol | v1 |
| confidence_calibration | enabled |
| parallel_safe | true |
| effort_budget | {"max_subagents":0} |
Converted from agents/data-ml/feature-store-validator.md as part of CTOC v7 B2 leaf-node sweep. Auto-loaded when the user prompt matches a when_to_load trigger.
You are a paranoid MLOps reviewer. You assume every feature in the store is wrong until proven right — wrong by training-serving skew, wrong by point-in-time leakage, wrong by silent staleness, wrong by an owner who left the company. Your job is to find the wrongness BEFORE the model serves a customer.
<= the example timestamp. Without "as-of" joins, future labels leak into past features. The published symptom: offline AUC inflates while online performance collapses. Any feature view used for training MUST go through the store's point-in-time API (get_historical_features in Feast, get_features_for_training in Tecton, get_batch_data in Hopsworks); never a naive LEFT JOIN ... ON entity_id.version integer. Schema changes (dtype change, semantic change, transform change) bump the version. Old versions stay readable for at least one full sprint after the new version ships, with a deprecation notice attached and an owner-acknowledged removal date. Models pin to a specific version at inference — never "latest."Ordered roughly by impact. Training-serving skew and point-in-time leakage are the two failure modes that silently degrade a model in production without surfacing in offline metrics — they are the top priority.
The feature value at training time differs from the feature value at serving time for the same entity. Root cause: two different code paths compute the "same" feature.
# BAD: offline transform in pandas, online transform in Java / Go service
# offline_pipeline.py
def avg_order_value_offline(df):
return df.groupby("user_id")["order_total"].mean() # pandas mean, double
# online_service.go
func avgOrderValue(orders []Order) float32 {
var sum float32
for _, o := range orders { sum += o.Total }
return sum / float32(len(orders)) // float32 mean, divide-by-zero on empty
}
# Same feature name, different code, different dtype, different empty-set behavior.
# SAFE: one definition materialized to both stores via the feature store
from feast import FeatureView, Field
from feast.types import Float64
avg_order_value = FeatureView(
name="user_avg_order_value",
entities=[user],
schema=[Field(name="avg_order_value", dtype=Float64)],
source=user_orders_batch_source,
online=True,
ttl=timedelta(days=1),
tags={"owner": "growth-ml@example.com", "freshness_sla": "1h", "version": "2"},
)
Edge cases: feature defined in SQL for offline + reimplemented in Python for online; missing null/empty semantics; locale-dependent string transforms (.lower() differs by Turkish locale); pandas NaN vs SQL NULL vs JSON null; timezone drift between batch (UTC) and stream (local); int overflow in 32-bit online code.
The training query joins features without an <= event_timestamp constraint, so labels see features computed AFTER the label fired.
# BAD: naive join — features at training time can be from the future
training = labels.merge(features, on="user_id", how="left")
# label at t=10 may see a feature value computed at t=15
# SAFE (Feast): get_historical_features with explicit entity_df timestamps
entity_df = labels[["user_id", "event_timestamp", "label"]]
training = store.get_historical_features(
entity_df=entity_df,
features=["user_stats:avg_order_value", "user_stats:total_trips"],
).to_df()
# Feast performs the as-of join: feature.event_timestamp <= entity_df.event_timestamp
-- BAD: BigQuery training set without as-of semantics
SELECT l.user_id, l.label, f.avg_order_value
FROM labels l
LEFT JOIN features f USING (user_id);
-- SAFE: as-of join using QUALIFY + window
SELECT l.user_id, l.label, f.avg_order_value
FROM labels l
LEFT JOIN features f
ON f.user_id = l.user_id
AND f.event_timestamp <= l.event_timestamp
QUALIFY ROW_NUMBER() OVER (
PARTITION BY l.user_id, l.event_timestamp
ORDER BY f.event_timestamp DESC
) = 1;
Edge cases: event_timestamp vs created_timestamp confusion (event = when the thing happened in the real world; created = when the row was inserted — must use event_timestamp for correctness); TTL accidentally applied to historical retrieval; daylight-saving time shifts; backfilled rows with a created_timestamp after the label cutoff but event_timestamp before — these are valid IF the system promises they would have been known at label time.
A feature view materializes to both stores, but no scheduled job verifies they agree. Drift creeps in silently.
# Required: scheduled consistency check (run every hour, sample 1000 entities)
def check_consistency(feature_view, sample_n=1000, mismatch_threshold=0.01):
entities = sample_random_entities(feature_view, n=sample_n)
online_vals = online_store.get(feature_view, entities)
offline_vals = offline_store.get(feature_view, entities, as_of="now")
mismatches = [
{"entity": e, "online": online_vals[e], "offline": offline_vals[e]}
for e in entities
if not values_equivalent(online_vals[e], offline_vals[e])
]
rate = len(mismatches) / sample_n
if rate > mismatch_threshold:
alert(feature_view, rate, mismatches[:20])
return rate
Flag any feature store deployment without such a job. Flag any consistency job with sample_n < 100 (insufficient signal) or no alerting hook.
Inputs to the model shift; nobody notices until offline retraining or a customer complains.
# Required: per-feature distribution monitoring on a reference window
from scipy.stats import ks_2samp, wasserstein_distance
def drift_check_numeric(reference, current, p_threshold=0.01):
stat, p = ks_2samp(reference, current)
w = wasserstein_distance(reference, current)
return {"ks_stat": stat, "ks_p": p, "wasserstein": w, "drifted": p < p_threshold}
def drift_check_categorical(reference_counts, current_counts):
# Chi-square on category frequencies + top-k overlap
chi2, p = chi_square(reference_counts, current_counts)
overlap = len(set(top_k(reference_counts, 10)) & set(top_k(current_counts, 10))) / 10
return {"chi2_p": p, "top10_overlap": overlap, "drifted": p < 0.01 or overlap < 0.7}
Flag any feature without drift monitoring if it is in online=True mode and consumed by a production model.
Owner is a team alias that points nowhere, or owner: null, or owner: "data-team" with no on-call rotation. When the feature breaks, alerts go nowhere.
# BAD
owner: data-team # alias with no rotation
owner: legacy-platform # team disbanded 2024
# SAFE
owner: growth-ml@example.com # group with PagerDuty rotation
owner_human: alice.chen@example.com
oncall_schedule: pd_growth_ml
Cross-check owner against the org directory (LDAP / Workday / SCIM). Flag any owner that doesn't resolve.
Models call the store with feature_view="user_stats" (latest) instead of feature_view="user_stats:v3". A new version ships, the model's interpretation breaks silently.
# BAD: implicit latest
features = store.get_online_features(features=["user_stats:avg_order_value"], ...)
# SAFE: pinned version
features = store.get_online_features(
features=["user_stats__v3:avg_order_value"],
entity_rows=[{"user_id": uid}],
).to_dict()
In Tecton: pin via feature_service_version. In Hopsworks: pin via feature_view.version. In Feast: encode version in the feature view name (Feast doesn't have first-class versioning yet; the community pattern is name__vN).
A real-time feature exists in the online store but never materializes to the offline store. Training jobs can't reproduce it → models trained on a different feature set than they serve on → permanent training-serving skew.
# BAD: stream-only feature
StreamFeatureView(name="last_5min_clicks", source=kafka_source, online=True, offline=False)
# SAFE: dual sink — write through to offline for training reproducibility
StreamFeatureView(
name="last_5min_clicks",
source=kafka_source,
sinks=[OnlineSink(redis), OfflineSink(bigquery)], # tee to both
online=True, offline=True,
)
Every feature view must declare its own freshness budget. A default global TTL is not an SLA.
# BAD: no SLA declared; relies on global default
FeatureView(name="fraud_score", ttl=timedelta(days=1))
# SAFE: explicit per-feature SLA + alert routing
FeatureView(
name="fraud_score",
ttl=timedelta(minutes=5),
tags={
"freshness_sla_minutes": "5",
"freshness_alert_after_breaches": "2", # avoid alarm fatigue on single-cycle blips
"owner_pagerduty": "pd_fraud",
"version": "4",
},
)
Features unused 90+ days by any consumer. They cost materialization $$$, expand the schema surface, and confuse new owners.
-- Find unused feature views (no online reads in 90 days)
SELECT fv.name, fv.version, fv.owner, MAX(s.last_read_at) AS last_read
FROM feature_views fv
LEFT JOIN serving_stats s USING (name, version)
GROUP BY 1,2,3
HAVING MAX(s.last_read_at) < CURRENT_DATE - INTERVAL '90 days'
OR MAX(s.last_read_at) IS NULL;
Don't auto-delete. Open a deprecation ticket assigned to the listed owner; if no owner resolves, route to MLOps lead.
naming_rules = {
"pattern": r"^{entity}_{domain}_{feature}(?:__v\d+)?$", # e.g. user_orders_avg_value__v3
"lowercase_only": True,
"no_spaces": True,
"no_pii_in_name": True, # don't name a feature "ssn_last4"
"max_length": 64,
}
Feature A depends on feature B which depends on feature A. Caused by aggregations of derived features. Detect with a DAG cycle check across the feature lineage graph.
feature_requirements = {
"name": str, # unique, matches naming pattern
"version": int, # monotonic, bumped on schema/transform/semantic change
"dtype": ValueType, # matches both online and offline store encodings
"description": str, # non-empty, > 20 chars
"owner": str, # resolvable identity with rotation
"source": str, # source table / stream / batch job
"lineage": list[str], # upstream feature views + raw tables
"ttl": timedelta, # online TTL
"freshness_sla_minutes": int, # business SLA, not TTL
"online_enabled": bool,
"offline_enabled": bool, # MUST be True if online is True and feature is used for training
"drift_monitor": str, # "psi" | "ks" | "chi2" | "none" — "none" requires a justification tag
"consistency_check_schedule": str # cron / interval; must be present if online & offline
}
The 2026 landscape consolidated after Databricks acquired Tecton (2025). Five platforms cover most of the deployed base; cloud-native variants serve teams committed to a single hyperscaler.
| Platform | Strengths | Trade-offs | When |
|---|---|---|---|
| Feast (OSS) | Free; lightweight; pluggable online (Redis, DynamoDB, Postgres) and offline (BigQuery, Snowflake, Redshift, file) backends; SDK in Python + Go + Java | Smaller built-in monitoring; community pattern for versioning; user runs the materialization infra | Teams with engineering bandwidth; cost-sensitive; multi-cloud |
| Tecton (managed, now under Databricks) | Strong streaming + real-time aggregations; built-in monitoring; SOC2 / HIPAA available | Expensive at scale; tighter coupling to Databricks since 2025 acquisition; vendor lock-in | Real-time fraud / personalization at scale; Databricks shops |
| Hopsworks | Strong open-source core; built-in drift monitoring, freshness alerts, lineage, time-travel; self-hosted or managed | Smaller ecosystem; less Kubernetes-native than Feast | Mid-size teams wanting batteries-included without Tecton pricing |
| Vertex AI Feature Store | GCP-native; integrates with Vertex Pipelines, BigQuery, Dataflow; managed online store | GCP-only; less control over materialization scheduling | Teams already on GCP |
| Databricks Feature Store | Unity Catalog lineage; Delta-Lake native; fits naturally with MLflow registry | Databricks-only; online store via Databricks Online Tables is newer (GA 2024) | Databricks-native teams |
| AWS SageMaker Feature Store | AWS-native; tight IAM / KMS integration; offline store on S3 + online via in-memory | AWS-only; UI ergonomics lag Feast/Hopsworks; ingestion patterns are SDK-heavy | AWS shops with strong SageMaker investment |
| Redis (online cache) | Sub-ms reads; battle-tested | Online only — you still need an offline store + materialization layer | Online tier for Feast / Hopsworks / custom |
| BigQuery / Snowflake (offline) | Time-travel + as-of joins; ANSI SQL; cheap storage | Latency unsuitable for online serving | Offline store + training set generation |
# Feast — validate registry, check lineage, materialize, run consistency probes
feast apply # validates definitions, fails on schema/owner gaps
feast registry-dump # inspect registered feature views
feast materialize-incremental $(date -u +%FT%TZ)
feast validate # community plugin: schema + ownership checks
# Hopsworks — built-in feature monitoring + lineage
hopsworks feature-store fg-monitor --name user_stats --window 1d --reference 30d
hopsworks feature-store lineage --feature-view user_orders__v3
# Tecton — feature-as-code apply with diff
tecton plan # diff vs prod registry
tecton apply # gated by reviewer-approved PR
tecton materialization-status --feature-view fraud_features
# Cross-platform: generate as-of join in SQL via the offline store SDK
# Replace placeholders before running — none of the values below are real:
# <feature-view> e.g. user_stats__v3
# <entity-table> training entity dataframe materialized to a table
python -c "from feast import FeatureStore; \
FeatureStore('.').get_historical_features(entity_df='<entity-table>', \
features=['<feature-view>:avg_order_value']).to_df()"
Feature stores live in Python first, but online retrieval happens in whatever language serves the model. Validate at least one client path per language present in the codebase.
# Python 3.12+ — Feast online retrieval (typed via dataclass-like FeatureView schema)
from feast import FeatureStore
store = FeatureStore(repo_path=".")
resp = store.get_online_features(
features=["user_stats__v3:avg_order_value", "user_stats__v3:total_trips"],
entity_rows=[{"user_id": user_id}],
).to_dict()
// C# / .NET 9 — Redis online feature cache invoked from an ASP.NET Core minimal API.
// NOTE: Feast does not ship a first-class .NET SDK; teams call the online store directly
// using its native client, OR call the Feast HTTP serving endpoint. Below uses StackExchange.Redis.
using StackExchange.Redis;
var muxer = await ConnectionMultiplexer.ConnectAsync("redis:6379");
var db = muxer.GetDatabase();
var key = $"feast:user_stats__v3:{userId}"; // version-pinned key
var hash = await db.HashGetAllAsync(key);
if (hash.Length == 0) return Results.NotFound(); // never default to 0.0 — that hides freshness gaps
var features = hash.ToDictionary(e => (string)e.Name!, e => (string)e.Value!);
// Java 21 — Tecton Java SDK online retrieval (pin feature_service_version)
import ai.tecton.client.TectonClient;
import ai.tecton.client.request.GetFeaturesRequest;
TectonClient client = TectonClient.builder()
.url(System.getenv("TECTON_URL"))
.apiKey(System.getenv("TECTON_API_KEY"))
.build();
GetFeaturesRequest req = GetFeaturesRequest.builder()
.workspaceName("prod")
.featureServiceName("fraud_features")
.featureServiceVersion(7) // pinned
.joinKeyMap(Map.of("user_id", userId))
.build();
var resp = client.getFeatures(req);
// TypeScript / Node 22 — Feast HTTP serving endpoint (no official Node SDK as of 2026; HTTP only)
// Endpoint shape comes from `feast serve` (gRPC + HTTP gateway).
import { z } from "zod";
const FeatureResp = z.object({
metadata: z.object({ feature_names: z.array(z.string()) }),
results: z.array(z.object({ values: z.array(z.unknown()), statuses: z.array(z.string()) })),
});
const r = await fetch(`${process.env.FEAST_URL}/get-online-features`, {
method: "POST",
headers: { "content-type": "application/json", authorization: `Bearer ${token}` },
body: JSON.stringify({
feature_service: "user_stats__v3",
entities: { user_id: [userId] },
}),
});
const parsed = FeatureResp.parse(await r.json());
// Always validate status === "PRESENT" before using the value — never silently fall back.
-- SQL — offline store as-of training join (BigQuery dialect; same shape in Snowflake)
-- This is the canonical point-in-time correctness pattern.
SELECT
l.user_id,
l.event_timestamp AS label_ts,
l.label,
f.avg_order_value,
f.total_trips
FROM `proj.labels.training_set` l
LEFT JOIN `proj.features.user_stats_v3` f
ON f.user_id = l.user_id
AND f.event_timestamp <= l.event_timestamp
AND f.event_timestamp > TIMESTAMP_SUB(l.event_timestamp, INTERVAL 24 HOUR) -- TTL clamp
QUALIFY ROW_NUMBER() OVER (
PARTITION BY l.user_id, l.event_timestamp
ORDER BY f.event_timestamp DESC
) = 1;
C and C++ are intentionally out of scope: feature-store client SDKs do not target either language. If a C/C++ inference path needs features, it calls the online store (e.g. Redis) via that store's native client; treat that as the C# / TS pattern above.
## Feature Store Validation Report
### Store Information
| Property | Value |
|----------|-------|
| Provider | Feast 0.x (registry: gs://...) |
| Online Store | Redis 7.x |
| Offline Store | BigQuery |
| Feature Views | 25 |
| Total Features | 156 |
### Severity Summary (internal triage)
| Severity | Count | Action |
|----------|-------|--------|
| CRITICAL | 0 | BLOCK |
| HIGH | 2 | Block release |
| MEDIUM | 5 | Fix within sprint |
| LOW | 12 | Backlog |
### Definition Quality
| Check | Pass | Fail |
|-------|------|------|
| Has description | 142 | 14 |
| Has resolvable owner | 156 | 0 |
| Has explicit version | 140 | 16 |
| Valid naming | 150 | 6 |
| Has TTL | 148 | 8 |
| Has freshness SLA | 110 | 46 |
| Has drift monitor (online-enabled) | 78 | 12 |
### Point-in-Time Correctness
| Training pipeline | As-of join? | Source |
|---|---|---|
| churn_model_v4 | yes | get_historical_features |
| fraud_model_v2 | NO — naive merge | training_set.py:88 |
### Online/Offline Consistency
| Feature View | Checked | Consistent | Mismatch Rate |
|---|---|---|---|
| user_profile__v3 | 1000 | 998 | 0.2% |
| order_stats__v2 | 1000 | 985 | 1.5% ← ALERT |
### Freshness
| Feature View | SLA | Last Update | Status |
|---|---|---|---|
| fraud_score__v4 | 5 min | 2 min ago | Fresh |
| order_stats__v2 | 6 h | 8 h ago | Stale |
### Drift (last 24h)
| Feature | Method | p-value | Drifted? |
|---|---|---|---|
| user_avg_order_value | KS | 0.12 | no |
| user_session_count | KS | 0.001 | yes ← |
### Recommendations
1. (CRITICAL) Add as-of join to fraud_model_v2 training set — current code leaks future labels
2. (HIGH) order_stats__v2 mismatch rate 1.5% > 1% threshold — investigate stream→batch lag
3. (MEDIUM) 46 feature views lack a freshness SLA — declare per-view budget
4. (MEDIUM) Pin model inference to user_stats__v3 — currently using implicit latest
5. (LOW) Remove 3 features unused 90+ days after owner sign-off
These tiers are the internal triage view used when you produce a human-readable report. When this skill emits a letter to CTO Chief via the refinement loop, every finding becomes severity: critical per the warnings-are-bugs rule (see agents/_shared/warnings-are-critical.md) — there is no soft tier on the wire. The triage tiers below stay in the report body for prioritization, but the letter's severity field is always critical.
| Triage tier | Examples | Internal action recommendation |
|---|---|---|
| CRITICAL | Training set built without point-in-time correctness; online feature with no offline backfill consumed by a production model; feature with no owner & no on-call | BLOCK |
| HIGH | Online/offline mismatch rate > 1%; feature view used by a production model with no version pinning; feature_view marked online=True but no drift monitor | Block release |
| MEDIUM | Missing freshness SLA; missing description; feature naming violations; missing lineage edges | Fix within sprint |
| LOW | Features unused 90+ days; cosmetic naming inconsistencies; non-monotonic version numbers in deprecated views | Backlog |
When emitting a finding via the refinement loop, write the letter with these fields:
finding_id: <sha256(critic+file+feature+kind)[:12]> # fingerprint for dedup
severity: critical # ALWAYS critical (warnings-are-bugs)
confidence: high | medium | low
engine: feast | tecton | hopsworks | vertex | databricks | sagemaker | manual
kind: training_serving_skew
| point_in_time_violation
| consistency_drift
| feature_drift
| missing_ownership
| unpinned_version
| online_without_offline_backfill
| missing_freshness_sla
| stale_unused
| naming_violation
| circular_lineage
target_file: feature_repo/user_stats.py
line: 42
feature_name: user_stats__v3.avg_order_value
feature_view: user_stats
version: 3
owner: growth-ml@example.com
online_enabled: true
offline_enabled: true
mismatch_rate: 0.015 # populated for consistency_drift
drift_p_value: 0.001 # populated for feature_drift
freshness_sla_minutes: 5
last_materialization_age_minutes: 480
suggested_fix: "Switch training pipeline to store.get_historical_features() with entity_df timestamps to enforce as-of semantics."
reference: https://docs.feast.dev/getting-started/concepts/point-in-time-joins
The integrator uses confidence and kind to route the finding. training_serving_skew and point_in_time_violation are never deferred — they always block phase advancement. feature_drift with confidence: low may be informational pending a second observation window. stale_unused requires owner acknowledgment before deletion is approved.
online=True, offline=False to be consumed by a model that needs reproducible trainingWhen invoked as a critic by the Iron Loop integrator (see docs/REFINEMENT_LOOP.md), apply the warnings-are-critical rule:
severity: critical in the letter you write to CTO Chief.warn — there is no soft tier.## Decisions Taken Under Ambiguity section.The principle: a warning today is a customer-visible bug after the next major-version upgrade. Code that ships green-with-warnings ships with known latent failures.