model-monitor

name	model-monitor
description	Set up and manage ML Observability for Snowflake Model Registry models. Use when: creating model monitors, tracking drift, viewing performance metrics, managing monitor lifecycle. Triggers: model monitor, ML observability, monitor model, drift detection, model performance, track predictions, add monitoring, enable monitoring, start monitoring, observability.
parent_skill	machine-learning

Model Monitor Operations

Intent Detection

Route based on user intent:

User Says	Route To
"create monitor", "set up monitoring", "track model", "monitor my model", "add monitoring", "enable monitoring", "start monitoring"	Workflow A: Create Model Monitor
"check drift", "view metrics", "model performance", "query metrics"	Workflow B: Query Monitor Metrics
"suspend monitor", "resume monitor", "add segment", "set baseline"	Workflow C: Manage Monitor
"monitor not working", "suspended", "refresh failing"	Workflow D: Troubleshoot Monitor

---

Prerequisites

Before using model monitors:

• Model must be registered in Snowflake Model Registry
• Model task must be tabular_binary_classification, tabular_regression, or tabular_multi_classification
• Source table with predictions and timestamps must exist
• snowflake-ml-python >= 1.7.1 required

---

Workflow A: Create Model Monitor

Step 0: Check for Recent Model Context

⚠️ IMPORTANT: Before asking questions, check if you have context from a recent session:

Context sources:

• From model-registry: Model name, version, database, schema, framework
• From spcs-inference: Model name, version, database, schema, service name

Context to look for:

• Model name and version
• Database and schema
• Source table (if predictions are being logged)
• Inference service name (if coming from SPCS)

If context exists:

• Use known model name/version - don't ask again
• Skip to Step 1, pre-filling the checklist with known values
• Only ask for missing required parameters (source table, warehouse, timestamp column, etc.)

If no context: Proceed to Step 1 and Step 2 normally.

Step 1: Parse Intent and Build Parameter Checklist

Analyze the user's request to determine which parameters to collect:

Feature Detection Table:

If user mentions...	Add to collection
"baseline", "drift", "PSI"	BASELINE
"segment", "subgroup", "slice"	SEGMENT_COLUMNS
"accuracy", "performance", "ground truth", "actual"	ACTUAL columns
"id column", "unique identifier", "row id"	ID_COLUMNS

Build checklist:

• Mandatory: monitor name, model, version, function, source, warehouse, refresh interval, aggregation window, timestamp, prediction columns
• BASELINE (if drift/baseline mentioned)
• SEGMENT_COLUMNS (if segment/subgroup mentioned)
• ACTUAL columns (if performance/accuracy mentioned)
• ID_COLUMNS (if unique identifier mentioned)

Step 2: Collect Mandatory Parameters

If recent context exists, only ask for parameters not already known:

I see you just registered model <MODEL_NAME> version <VERSION>. To set up monitoring, I need:

1. Monitor name: [identifier for the monitor]
2. Function name: [e.g., 'predict']
3. Source table: [table with predictions, fully qualified]
4. Warehouse: [for monitor compute]
5. Refresh interval: [e.g., '1 day', '6 hours', min: '60 seconds']
6. Aggregation window: [days only, e.g., '1 day', '7 days']
7. Timestamp column: [column name, must be TIMESTAMP_NTZ]
8. Prediction column(s): [score or class columns]

If no context, ask for all parameters:

To create your model monitor, I need the following information:

1. Monitor name: [identifier for the monitor]
2. Model name: [must be in Model Registry]
3. Model version: [version to monitor]
4. Function name: [e.g., 'predict']
5. Source table: [fully qualified: DATABASE.SCHEMA.TABLE]
6. Warehouse: [for monitor compute]
7. Refresh interval: [e.g., '1 day', '6 hours', min: '60 seconds']
8. Aggregation window: [days only, e.g., '1 day', '7 days']
9. Timestamp column: [column name, must be TIMESTAMP_NTZ]
10. Prediction column(s): [score or class columns]

⚠️ STOP: Wait for user response.

Step 3: Collect Feature-Specific Parameters

Based on Step 1 analysis, ask for additional parameters:

If BASELINE needed:

For drift detection, I also need:
- Baseline table: [fully qualified: DATABASE.SCHEMA.TABLE]

If SEGMENT_COLUMNS needed:

For segmentation, I also need:
- Segment column(s): [column names, must be STRING type, max 5, <25 unique values recommended, no special characters in values]

If ACTUAL columns needed:

For performance metrics, I also need:
- Actual/ground truth column(s): [score or class columns]

⚠️ STOP: Wait for user response (only if additional parameters needed).

Step 4: Validate Prerequisites

-- Verify model and version exist
SHOW VERSIONS IN MODEL <DATABASE>.<SCHEMA>.<MODEL_NAME>;

If model doesn't exist: Direct user to register model first using ../model-registry/SKILL.md.

-- Check source table columns
DESCRIBE TABLE <SOURCE_TABLE>;

Verify:

• Timestamp column exists (must be TIMESTAMP_NTZ)
• Prediction columns exist:
  • Binary classification/regression: NUMBER type
  • Multi-class classification: STRING type
• Actual columns (if provided): same type rules as prediction columns
• Segment columns (if provided): must be STRING type

Step 5: Generate and Execute SQL

Build CREATE MODEL MONITOR statement with collected parameters:

CREATE MODEL MONITOR <MONITOR_NAME> WITH
    MODEL = <MODEL_NAME>
    VERSION = '<VERSION_NAME>'
    FUNCTION = '<FUNCTION_NAME>'
    SOURCE = <SOURCE_TABLE>
    WAREHOUSE = <WAREHOUSE_NAME>
    REFRESH_INTERVAL = '<REFRESH_INTERVAL>'
    AGGREGATION_WINDOW = '<AGGREGATION_WINDOW>'
    TIMESTAMP_COLUMN = <TIMESTAMP_COL>
    PREDICTION_SCORE_COLUMNS = ('<PRED_COL>')
    -- Include if ACTUAL columns collected:
    ACTUAL_SCORE_COLUMNS = ('<ACTUAL_COL>')
    -- Include if BASELINE collected:
    BASELINE = <BASELINE_TABLE>
    -- Include if SEGMENT_COLUMNS collected:
    SEGMENT_COLUMNS = ('<SEGMENT_COL_1>', '<SEGMENT_COL_2>');

⚠️ MANDATORY: Present the generated SQL (with only the relevant clauses) to user and wait for approval before executing.

Step 6: Verify Monitor Created

-- List all monitors in schema
SHOW MODEL MONITORS IN SCHEMA <DATABASE>.<SCHEMA>;

-- Filter by name pattern
SHOW MODEL MONITORS LIKE '%<PATTERN>%' IN SCHEMA <DATABASE>.<SCHEMA>;

-- Check specific monitor details
DESC MODEL MONITOR <MONITOR_NAME>;

Verify monitor_state is ACTIVE (not SUSPENDED, PARTIALLY_SUSPENDED, or UNKNOWN).

⚠️ STOP: Confirm monitor is running before proceeding.

Suggested Next Actions

Present context-aware options based on what was configured:

If baseline was NOT configured:

Your model monitor is active! To enable drift detection:
  - "Set a baseline for drift detection"

If actual columns were NOT configured:

To track model accuracy, you can add ground truth data later:
  - "Add actual columns to my monitor" (requires recreating monitor)

Always show:

View your monitor: Snowsight → AI&ML → Models → Select Model → Monitors
Query metrics: "Show me metrics for my monitor"

---

Workflow B: Query Monitor Metrics

Step 1: Identify Monitor and Metric Type

Ask user:

What metrics would you like to view?

1. Drift metrics (PSI, distribution shifts)
2. Performance metrics (accuracy, precision, recall, F1, MAE, RMSE)
3. Statistical metrics (counts, nulls, distributions)

⚠️ STOP: Wait for user response.

Step 2: Query Metrics

Drift Metrics:

SELECT * 
FROM TABLE(MODEL_MONITOR_DRIFT_METRIC(
    '<MONITOR_NAME>',
    '<METRIC_NAME>',        -- e.g., 'PSI', 'KL_DIVERGENCE'
    '<COLUMN_NAME>',        -- feature column to check drift
    'DAY',                  -- granularity
    '<START_TIME>'::TIMESTAMP_NTZ,
    '<END_TIME>'::TIMESTAMP_NTZ
));

Performance Metrics:

SELECT * 
FROM TABLE(MODEL_MONITOR_PERFORMANCE_METRIC(
    '<MONITOR_NAME>',
    '<METRIC_NAME>',        -- 'ACCURACY', 'PRECISION', 'RECALL', 'F1', 'MAE', 'RMSE'
    'DAY',
    '<START_TIME>'::TIMESTAMP_NTZ,
    '<END_TIME>'::TIMESTAMP_NTZ
));

Statistical Metrics:

SELECT * 
FROM TABLE(MODEL_MONITOR_STAT_METRIC(
    '<MONITOR_NAME>',
    '<METRIC_NAME>',        -- 'COUNT', 'NULL_COUNT', etc.
    'DAY',
    '<START_TIME>'::TIMESTAMP_NTZ,
    '<END_TIME>'::TIMESTAMP_NTZ
));

Query Metrics for Specific Segment:

SELECT * 
FROM TABLE(MODEL_MONITOR_DRIFT_METRIC(
    '<MONITOR_NAME>',
    'PSI',
    '<COLUMN_NAME>',
    'DAY',
    '<START_TIME>'::TIMESTAMP_NTZ,
    '<END_TIME>'::TIMESTAMP_NTZ,
    '{"SEGMENTS": [{"column": "<SEGMENT_COL>", "value": "<SEGMENT_VALUE>"}]}'
));

Step 3: Present Results

Format results clearly showing:

• Time period analyzed
• Metric values and trends
• Any concerning drift or performance degradation

Suggest next actions based on results:

• High drift → Investigate data pipeline, consider retraining
• Low performance → Review model, check for concept drift
• Missing data → Check source table updates

---

Workflow C: Manage Monitor

Suspend/Resume Monitor

-- Suspend monitoring
ALTER MODEL MONITOR <MONITOR_NAME> SUSPEND;

-- Resume monitoring
ALTER MODEL MONITOR <MONITOR_NAME> RESUME;

Set Monitor Properties

Set one or more properties in a single statement (each is optional):

ALTER MODEL MONITOR <MONITOR_NAME> SET
    BASELINE = '<BASELINE_TABLE_NAME>'
    REFRESH_INTERVAL = '<REFRESH_INTERVAL>'
    WAREHOUSE = <WAREHOUSE_NAME>;

Examples:

-- Set baseline only
ALTER MODEL MONITOR <MONITOR_NAME> SET BASELINE = '<BASELINE_TABLE>';

-- Set refresh interval only
ALTER MODEL MONITOR <MONITOR_NAME> SET REFRESH_INTERVAL = '6 hours';

-- Set warehouse only
ALTER MODEL MONITOR <MONITOR_NAME> SET WAREHOUSE = <NEW_WAREHOUSE>;

-- Set multiple properties at once
ALTER MODEL MONITOR <MONITOR_NAME> SET
    REFRESH_INTERVAL = '6 hours'
    WAREHOUSE = <NEW_WAREHOUSE>;

Add/Remove Segment Columns

-- Add segment (max 5 segments, must be STRING type)
ALTER MODEL MONITOR <MONITOR_NAME> ADD SEGMENT_COLUMN = '<COLUMN_NAME>';

-- Remove segment
ALTER MODEL MONITOR <MONITOR_NAME> DROP SEGMENT_COLUMN = '<COLUMN_NAME>';

---

Drop Model Monitor

⚠️ MANDATORY: Before dropping, confirm with user:

Are you sure you want to delete monitor <MONITOR_NAME>?

This cannot be undone. All monitor history and metrics will be lost.

Type "yes" to confirm:

⚠️ STOP: Wait for explicit confirmation before proceeding.

If confirmed:

DROP MODEL MONITOR <MONITOR_NAME>;

---

Workflow D: Troubleshoot Monitor

Step 1: Check Monitor Status

DESC MODEL MONITOR <MONITOR_NAME>;

Check monitor_state:

• ACTIVE - Operating correctly
• SUSPENDED - Monitoring paused (manual or after 5 failures)
• PARTIALLY_SUSPENDED - One underlying table stopped refreshing
• UNKNOWN - State cannot be identified

Additional fields for debugging:

• aggregation_status - Current aggregation state
• aggregation_last_error - Error details if suspended

Step 2: Common Issues and Solutions

Issue	Cause	Solution
SUSPENDED	Manual suspend or 5 consecutive refresh failures	Fix root cause, then ALTER MODEL MONITOR ... RESUME
PARTIALLY_SUSPENDED	One underlying table stopped refreshing	Check DESC for aggregation_last_error, fix issue, then RESUME
UNKNOWN	State cannot be determined	Check DESC for errors, may need to recreate monitor
No drift metrics	No baseline set	Set baseline with ALTER MODEL MONITOR ... SET BASELINE
Missing performance metrics	No actual columns	Re-create monitor with ACTUAL_SCORE_COLUMNS
Segment query fails	Invalid segment value	Check segment column has expected values, case sensitive
Invalid data errors	NULLs, NaNs, or out-of-range values	Clean source data, remove invalid rows

Step 3: Resume After Fixing

-- After fixing root cause
ALTER MODEL MONITOR <MONITOR_NAME> RESUME;

-- Verify it's running
DESC MODEL MONITOR <MONITOR_NAME>;

---

CREATE MODEL MONITOR Parameters

Mandatory Parameters (Always Required)

Parameter	Type	Description
MONITOR_NAME	identifier	Name for the monitor
MODEL	identifier	Model name (same schema as monitor)
VERSION	string	Model version name
FUNCTION	string	Function name (e.g., 'predict')
SOURCE	identifier	Table/view with predictions (fully qualified)
WAREHOUSE	identifier	Warehouse for compute
REFRESH_INTERVAL	string	How often to refresh (min: '60 seconds', e.g., '1 day')
AGGREGATION_WINDOW	string	Aggregation period (days only, e.g., '1 day')
TIMESTAMP_COLUMN	identifier	Timestamp column (must be TIMESTAMP_NTZ)
PREDICTION_*_COLUMNS	string list	At least one of: PREDICTION_CLASS_COLUMNS or PREDICTION_SCORE_COLUMNS

Feature-Triggered Parameters (Ask When User Mentions Feature)

Parameter	Trigger Keywords	Type	Description
BASELINE	"baseline", "drift", "PSI"	identifier	Baseline table for drift detection
SEGMENT_COLUMNS	"segment", "subgroup", "slice"	string list	Columns for segmentation (max 5, STRING type, <25 unique values recommended)
ACTUAL_CLASS_COLUMNS	"accuracy", "performance", "ground truth", "actual"	string list	Ground truth class columns
ACTUAL_SCORE_COLUMNS	"accuracy", "performance", "ground truth", "actual"	string list	Ground truth score columns

Optional Parameters (Only If User Explicitly Requests)

Parameter	Type	Description
ID_COLUMNS	string list	Columns that uniquely identify each row in source data
CUSTOM_METRIC_COLUMNS	string list	Additional numeric columns to track

---

Usage Notes

• Column uniqueness: Each column can only be used in one parameter (e.g., an ID column cannot also be a prediction column), except for segment columns which can overlap with other columns as long as they are STRING type
• Single output only: Multiple-output models not supported; prediction/actual column arrays must have exactly one element
• Segment values: Case sensitive, special characters not supported in segment queries

---

Monitor Status Commands

List Monitors

-- All monitors in current schema
SHOW MODEL MONITORS;

-- Filter by pattern
SHOW MODEL MONITORS LIKE '%mymodel%';

-- Specific scope
SHOW MODEL MONITORS IN SCHEMA <DATABASE>.<SCHEMA>;
SHOW MODEL MONITORS IN DATABASE <DATABASE>;
SHOW MODEL MONITORS IN ACCOUNT;

Key output column: monitor_state (ACTIVE, SUSPENDED, PARTIALLY_SUSPENDED, UNKNOWN)

Describe Monitor

DESC MODEL MONITOR <MONITOR_NAME>;

Key output columns for debugging:

• monitor_state - Current state
• aggregation_status - Aggregation state details
• aggregation_last_error - Error message if suspended
• columns - JSON with all configured columns

---

Supported Metrics

Drift Metrics:

• PSI (Population Stability Index)
• KL_DIVERGENCE (Kullback-Leibler Divergence)

Performance Metrics (Classification):

• ACCURACY, PRECISION, RECALL, F1
• AUC, LOG_LOSS

Performance Metrics (Regression):

• MAE (Mean Absolute Error)
• RMSE (Root Mean Square Error)
• MAPE (Mean Absolute Percentage Error)

Statistical Metrics:

• COUNT, NULL_COUNT
• MEAN, STDDEV, MIN, MAX

---

Limitations

• Max 250 monitors per account
• Max 500 features monitored per monitor
• Max 5 segment columns per monitor
• Segment columns must be STRING type, <25 unique values recommended
• Segment values are case sensitive, no special characters supported
• Multiple-output models not supported (arrays must have one element)
• Supports regression, binary classification, and multi-class classification
• Monitor must be in same schema as model
• Cannot change model/source after creation (must drop and recreate)

---

Output

• Model monitor created and tracking predictions
• Metrics available via SQL functions or Snowsight UI
• Alerts can be configured using Snowflake alerting features