data-analysis

name	data-analysis
description	DuckDB-powered analytical engine for large data dumps (JSON, CSV, Parquet). Ingest → Profile → Query → File insights.
argument-hint	analyze <file_path>
allowed-tools	["Bash","FileWrite"]
auto-invoke	false
model	default
user-invocable	true
context_trigger	["*.json > 10MB","*.csv > 10MB","data dump","analyze this data","large dataset"]
who	Athena agent performing data analysis
what	Ingest, profile, query, and extract insights from large structured datasets
when	User provides a large data file (JSON/CSV/Parquet) or asks for data analysis
where	Local filesystem, DuckDB in-process
why	Eliminates ad-hoc Python one-shots for large data analysis. Provides reusable, cached, SQL-queryable analytical backend.
how	DuckDB embedded OLAP engine + Parquet columnar storage + auto-filing to case studies

Data Analysis Skill (DuckDB Engine)

Wraps the Athena Data Engine (data_engine.py) for structured data analysis on large files.

Triggers

• User provides a JSON, CSV, or Parquet file for analysis
• "Analyze this data", "What's in this file", "Run some numbers on"
• Any file > 10MB that needs analytical queries
• /analyze workflow invocation

Dependencies

pip install duckdb

Core Scripts

Script	Purpose
.agent/scripts/data_engine.py	Core DuckDB wrapper — ingest, convert, query, profile
.agent/scripts/auto_file_insights.py	Auto-file extracted insights as case studies

Pipeline

Phase 1: Ingest

python3 .agent/scripts/data_engine.py ingest /path/to/data.json

This will:

1. Auto-detect format (JSON/CSV/Parquet)
2. For Telegram exports: flatten nested text fields, extract metadata
3. Convert to Parquet with ZSTD compression (cached alongside the original)
4. Print schema, row count, nulls, date range, value distributions

Cache behavior: If a Parquet cache already exists and is newer than the source file, ingestion is skipped and the cache is used directly (instant).

Phase 2: Query

python3 .agent/scripts/data_engine.py query /path/to/.athena_cache/parquet/data.parquet \
    "SELECT COUNT(*) FROM data WHERE text LIKE '%math%'"

The Parquet file is registered as table data. Use standard SQL.

Common patterns:

-- Row count
SELECT COUNT(*) FROM data

-- Date range
SELECT MIN(date), MAX(date) FROM data


 column_name, ()  cnt
 data
  column_name
  cnt 
LIMIT 


 , text  data
 text ILIKE 
LIMIT 


 strftime(::, )  , ()  volume
 data
  
  


 regexp_extract(text, , )::  rate, ()  cnt
 data
 rate  
  rate
  cnt