dr-anomalies

# Datarails Anomaly Detection Automated anomaly detection for Finance OS tables - find data quality issues, outliers, and suspicious patterns. ## Workflow ### Step 1: Verify Authentication If any tool call fails with an authentication or connection error, guide the user to connect via the Connectors UI ("+" > Connectors > Datarails > Connect). ### Step 2: Run Detection 1. Get table schema for context 2. Run `detect_anomalies` for comprehensive analysis 3. Optionally run `profile_numeric_fields` and `profile_categorical_fields` for deeper stats 4. If specific anomalies need investigation, use `get_records_by_filter` to fetch examples ### Step 3: Present Findings Organize findings by severity: - 🔴 **Critical** - Requires immediate attention - 🟠 **High** - Should be addressed soon - 🟡 **Medium** - Worth investigating - 🟢 **Low** - Minor issues or informational ## Arguments | Argument | Description | |----------|-------------| | `<table_id>` | Required - the table to analyze | | `--severity <level>` | Filter to specific severity (critical, high, medium, low) | | `--type <type>` | Filter to specific anomaly type | ## Anomaly Types Detected | Type | Description | |------|-------------| | `outliers` | Numeric values beyond 3 standard deviations | | `missing` | Unexpected NULL values or patterns | | `duplicates` | Potential duplicate records | | `temporal` | Date/time anomalies (gaps, future dates) | | `categorical` | Rare values, unexpected categories | | `referential` | Foreign key or relationship issues | ## Example Interactions **User: "/dr-anomalies 11442"** ``` 🔍 Anomaly Detection: GL Transactions (ID: 11442) ═══════════════════════════════════════════════════════════ Scanned 125,432 records | Found 47 anomalies 🔴 CRITICAL (3 findings) ─────────────────────────────────────────────────────────── 1. DUPLICATE TRANSACTIONS • 23 potential duplicate records detected • Same amount, date, and vendor within 1 minute • Records: [45231, 45232], [67892, 67893], ... 💡 Recommendation: Review for accidental double-entry 📋 Query: /dr-query 11442 "transaction_id IN (45231, 45232)" 2. FUTURE-DATED TRANSACTIONS • 5 transactions with posting_date > today • Dates range from 2024-02-15 to 2024-12-31 💡 Recommendation: Verify if these are planned entries 📋 Query: /dr-query 11442 "posting_date > '2024-01-20'" 3. NEGATIVE INVENTORY QUANTITIES • 8 records with quantity < 0 • Min value: -500 (record 89234) 💡 Recommendation: Check if returns are properly coded 🟠 HIGH (12 findings) ─────────────────────────────────────────────────────────── 4. AMOUNT OUTLIERS • 127 transactions beyond normal range • 115 above $500,000 (expected max ~$250,000) • 12 below -$100,000 (expected min ~-$50,000) 💡 Recommendation: Verify large transactions are approved 5. HIGH NULL RATE: vendor_name • 2,341 records (1.87%) missing vendor_name • But vendor_id is present 💡 Recommendation: Join with vendor master to populate ... 🟡 MEDIUM (18 findings) ─────────────────────────────────────────────────────────── 12. RARE CATEGORY VALUES • department contains 3 values appearing < 10 times • Values: "TEST", "MIGRATION", "UNKNOWN" 💡 Recommendation: Standardize or reclassify ... 🟢 LOW (14 findings) ─────────────────────────────────────────────────────────── 35. TRAILING WHITESPACE • account_code has 45 values with trailing spaces 💡 Recommendation: Trim during ETL ═══════════════════════════════════════════════════════════ 📊 SUMMARY ═══════════════════════════════════════════════════════════ | Severity | Count | Action | |----------|-------|---------------------------| | Critical | 3 | Investigate immediately | | High | 12 | Address this week | | Medium | 18 | Plan for remediation | | Low | 14 | Fix during maintenance | Data Quality Score: 87/100 ⚠️ Primary concerns: Duplicates, Future dates, Amount outliers ``` **User: "/dr-anomalies 11442 --severity critical"** ``` 🔴 Critical Anomalies: GL Transactions Found 3 critical issues requiring immediate attention: 1. DUPLICATE TRANSACTIONS (23 records) ... 2. FUTURE-DATED TRANSACTIONS (5 records) ... 3. NEGATIVE INVENTORY QUANTITIES (8 records) ... ``` **User: "/dr-anomalies 11442 --type outliers"** ``` 📊 Outlier Analysis: GL Transactions Analyzed 8 numeric fields for statistical outliers (|z| > 3) amount: 127 outliers ├── Above 3σ: 115 records │ ├── Max: $8,750,000 (z=12.4) │ ├── Sample: [45231: $2.1M], [67892: $1.8M], [89234: $1.5M] │ └── Pattern: Mostly Q4 entries (82%) └── Below -3σ: 12 records ├── Min: -$1,250,000 (z=-8.2) └── Sample: [12345: -$800K], [23456: -$650K] quantity: 23 outliers ├── All above 3σ (high quantities) ├── Max: 10,000 (z=5.1) └── 18 of 23 are from department="Warehouse" unit_cost: 45 outliers ... ``` ## Investigation Workflow When anomalies are detected: 1. **Review findings** - Understand the scope and patterns 2. **Fetch examples** - Use `/dr-query` to see actual records 3. **Verify business rules** - Some "anomalies" may be valid 4. **Document decisions** - Note which are false positives 5. **Create remediation plan** - Prioritize by severity ## Related Skills - `/dr-profile` - Detailed field statistics - `/dr-query` - Fetch specific records - `/dr-tables` - Understand table structure