data-cost-optimizer

license	Apache-2.0
name	data-cost-optimizer
description	Warehouse cost reduction, auto-scaling, query optimization, and lifecycle policies for data infrastructure. Activate on: data cost, warehouse credits, cost reduction, auto-scaling, lifecycle policy, cold storage, cost monitoring, resource optimization. NOT for: query performance tuning (use data-warehouse-optimizer), batch job optimization (use batch-processing-optimizer).
allowed-tools	Read,Write,Edit,Bash(npm:*,npx:*,python:*,snowsql:*,bq:*,aws:*)
category	Data & Analytics
tags	["cost-optimization","warehouse-cost","auto-scaling","lifecycle-policy","finops"]
pairs-with	[{"skill":"data-warehouse-optimizer","reason":"Query optimization directly reduces compute costs"},{"skill":"lakehouse-architect","reason":"Lakehouse storage is dramatically cheaper than warehouse storage"},{"skill":"airflow-dag-orchestrator","reason":"Scheduling optimization reduces compute costs"}]

Data Cost Optimizer

Reduce data infrastructure costs through warehouse right-sizing, storage lifecycle policies, query optimization, and FinOps practices.

Activation Triggers

Activate on: "data cost", "warehouse credits", "cost reduction", "auto-scaling", "lifecycle policy", "cold storage", "cost monitoring", "Snowflake credits", "BigQuery slots", "FinOps"

NOT for: Query performance tuning → data-warehouse-optimizer | Batch job resource sizing → batch-processing-optimizer | Infrastructure provisioning → relevant DevOps skill

Quick Start

1. Audit current spend — break down costs by warehouse/project, compute vs storage, team/department
2. Identify waste — idle warehouses, full table scans, duplicate data, over-provisioned resources
3. Implement auto-scaling — auto-suspend idle warehouses, auto-scale for concurrency peaks
4. Apply lifecycle policies — move old data to cheaper storage tiers automatically
5. Set budgets and alerts — per-team cost budgets with alerts at 80% and 100%

Core Capabilities

Domain	Technologies
Snowflake	Resource monitors, auto-suspend, warehouse sizing, credit tracking
BigQuery	Slot reservations, flat-rate vs on-demand, BI Engine, editions
Storage	S3 lifecycle (Standard → IA → Glacier), GCS Nearline/Coldline
Monitoring	Snowflake Account Usage, BigQuery INFORMATION_SCHEMA, Cost Explorer
FinOps	Kubecost, Datadog Cloud Cost, custom dashboards

Architecture Patterns

Snowflake Cost Control Framework

-- 1. Resource monitor: alert at 80%, suspend at 100%
CREATE RESOURCE MONITOR monthly_budget
  WITH CREDIT_QUOTA = 5000
  FREQUENCY = MONTHLY
  START_TIMESTAMP = IMMEDIATELY
  TRIGGERS
    ON 80  DO NOTIFY
       DO SUSPEND;

 WAREHOUSE analytics_wh  RESOURCE_MONITOR  monthly_budget;


 WAREHOUSE analytics_wh 
  AUTO_SUSPEND            
  AUTO_RESUME  
  MIN_CLUSTER_COUNT  
  MAX_CLUSTER_COUNT        
  SCALING_POLICY  ; 



  query_id,
  user_name,
  warehouse_name,
  total_elapsed_time    seconds,
  bytes_scanned  ()  gb_scanned,
  credits_used_cloud_services
 snowflake.account_usage.query_history
 start_time  DATEADD(, , )
  credits_used_cloud_services 
LIMIT ;