ti-ioc-extraction

# IOC Extraction from Unstructured Text Extract Indicators of Compromise from threat intelligence reports, incident logs, paste dumps, and raw text. Normalize, deduplicate, and format for downstream consumption (MISP, STIX/TAXII feeds, YARA rules, blocklists). ## Quick Reference ```bash # Install tools pip install ioc-finder ioc_fanger stix2 pymisp # One-shot extraction from a report file python3 -c " from ioc_finder import find_iocs text = open('/workspace/report.txt').read() iocs = find_iocs(text) for k,v in iocs.items(): if v: print(f'{k}: {v}') " # Refang a defanged IOC list python3 -c " from ioc_fanger import fang text = open('/workspace/defanged_iocs.txt').read() print(fang(text)) " > /workspace/refanged_iocs.txt # Quick regex extraction (no deps) grep -oP '\b[0-9a-fA-F]{32}\b' /workspace/report.txt | sort -u > /workspace/md5_hashes.txt grep -oP '\b[0-9a-fA-F]{64}\b' /workspace/report.txt | sort -u > /workspace/sha256_hashes.txt grep -oP '\b(?:[0-9]{1,3}\.){3}[0-9]{1,3}\b' /workspace/report.txt | sort -u > /workspace/ips.txt ``` ## MITRE ATT&CK Mapping | Technique | ID | Relevance | |---|---|---| | Gather Victim Network Information | T1590 | Extracted IPs, domains, and URLs map victim-facing infrastructure | | Gather Victim Host Information | T1592 | Extracted file hashes, registry keys, and file paths indicate host-level artifacts | | Phishing | T1566 | Extracted email addresses and sender domains from phishing report IOCs | | Indicator Removal | T1070 | Awareness of IOC types attackers attempt to obscure or rotate | ## 1. Regex-Based IOC Extraction When `ioc-finder` is unavailable or you need precise control, use these regex patterns: ```bash # MD5 hashes (32 hex chars, standalone word boundary) grep -oP '\b[0-9a-fA-F]{32}\b' /workspace/<REPORT> | sort -u > /workspace/iocs/md5.txt # SHA1 hashes (40 hex chars) grep -oP '\b[0-9a-fA-F]{40}\b' /workspace/<REPORT> | sort -u > /workspace/iocs/sha1.txt # SHA256 hashes (64 hex chars) grep -oP '\b[0-9a-fA-F]{64}\b' /workspace/<REPORT> | sort -u > /workspace/iocs/sha256.txt # IPv4 addresses grep -oP '\b(?:(?:25[0-5]|2[0-4]\d|[01]?\d\d?)\.){3}(?:25[0-5]|2[0-4]\d|[01]?\d\d?)\b' \ /workspace/<REPORT> | sort -u > /workspace/iocs/ipv4.txt # IPv6 addresses (simplified — catches common forms) grep -oiP '(?:[0-9a-f]{1,4}:){7}[0-9a-f]{1,4}|(?:[0-9a-f]{1,4}:){1,7}:|::(?:[0-9a-f]{1,4}:){0,5}[0-9a-f]{1,4}' \ /workspace/<REPORT> | sort -u > /workspace/iocs/ipv6.txt # Domain names (basic, excludes common FPs) grep -oP '(?:[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?\.)+(?:com|net|org|io|ru|cn|xyz|top|info|biz|cc|tk|ml|ga|cf|pw|buzz|su|onion)\b' \ /workspace/<REPORT> | sort -u > /workspace/iocs/domains.txt # URLs (http/https/ftp) grep -oP 'https?://[^\s"<>\])+' /workspace/<REPORT> | sort -u > /workspace/iocs/urls.txt # Email addresses grep -oP '[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}' \ /workspace/<REPORT> | sort -u > /workspace/iocs/emails.txt # CVE identifiers grep -oP 'CVE-\d{4}-\d{4,7}' /workspace/<REPORT> | sort -u > /workspace/iocs/cves.txt # MITRE ATT&CK technique IDs grep -oP 'T\d{4}(?:\.\d{3})?' /workspace/<REPORT> | sort -u > /workspace/iocs/mitre_ids.txt ``` ### False Positive Reduction ```bash # Filter out private/reserved IPs from extracted list grep -vP '^(10\.|172\.(1[6-9]|2\d|3[01])\.|192\.168\.|127\.|0\.|255\.)' \ /workspace/iocs/ipv4.txt > /workspace/iocs/ipv4_public.txt # Filter out known benign domains (Google, Microsoft, etc.) grep -viP '(google\.com|microsoft\.com|windowsupdate\.com|apple\.com|amazonaws\.com|cloudflare\.com)$' \ /workspace/iocs/domains.txt > /workspace/iocs/domains_filtered.txt # Remove hash-like strings that are too short or all-zero grep -vP '^0{32,}$' /workspace/iocs/md5.txt | grep -vP '^(.)\1{31,}$' > /workspace/iocs/md5_clean.txt ``` ## 2. Defanging and Refanging Threat reports often defang IOCs to prevent accidental clicks. You must refang before lookups. ```bash # Refang with ioc_fanger (Python) python3 -c " from ioc_fanger import fang import sys text = sys.stdin.read() print(fang(text)) " < /workspace/defanged_report.txt > /workspace/refanged_report.txt # Manual refang with sed (no deps) sed -e 's/\[.\]/./g' \ -e 's/hxxp/http/g' \ -e 's/hxxps/https/g' \ -e 's/ dot /./g' \ -e 's/\[at\]/@/g' \ -e 's/{dot}/./g' \ /workspace/defanged_report.txt > /workspace/refanged_report.txt # Defang IOCs for safe sharing python3 -c " from ioc_fanger import defang import sys text = sys.stdin.read() print(defang(text)) " < /workspace/iocs/urls.txt > /workspace/iocs/urls_defanged.txt # Manual defang with sed sed -e 's/\./[.]/g' \ -e 's|http://|hxxp://|g' \ -e 's|https://|hxxps://|g' \ -e 's/@/[at]/g' \ /workspace/iocs/urls.txt > /workspace/iocs/urls_defanged.txt ``` ### Common Defang Patterns to Recognize | Original | Defanged Variants | |---|---| | `192.168.1.1` | `192[.]168[.]1[.]1`, `192.168.1 .1`, `192(.)168(.)1(.)1` | | `http://evil.com` | `hxxp://evil[.]com`, `hxxp://evil(.)com`, `http[:]//evil.com` | | `user@evil.com` | `user[at]evil[.]com`, `user[@]evil.com` | ## 3. Programmatic Extraction with ioc-finder ```bash # Full extraction pipeline python3 << 'PYEOF' from ioc_finder import find_iocs import json, sys text = open('/workspace/<REPORT>').read() iocs = find_iocs(text) # Filter to populated categories only results = {k: list(set(v)) for k, v in iocs.items() if v} with open('/workspace/iocs/extracted.json', 'w') as f: json.dump(results, f, indent=2, default=str) # Summary for category, items in sorted(results.items()): print(f" {category}: {len(items)} unique") print(f"\nTotal categories with hits: {len(results)}") PYEOF # ioc-finder extracts these categories automatically: # - md5s, sha1s, sha256s, sha512s, ssdeeps # - ipv4s, ipv6s, urls, domains, email_addresses # - asns, cves, registry_key_paths, file_paths # - mac_addresses, bitcoin_addresses, xmpp_addresses # - google_adsense_ids, google_analytics_ids ``` ## 4. Bulk Hash Lookup ```bash # VirusTotal bulk lookup (requires API key) VT_API_KEY="<API_KEY>" while IFS= read -r hash; do result=$(curl -s "https://www.virustotal.com/api/v3/files/$hash" \ -H "x-apikey: $VT_API_KEY") detections=$(echo "$result" | python3 -c " import sys, json d = json.load(sys.stdin) stats = d.get('data',{}).get('attributes',{}).get('last_analysis_stats',{}) print(f\"{stats.get('malicious',0)}/{sum(stats.values())}\")" 2>/dev/null) echo "$hash $detections" >> /workspace/iocs/vt_results.txt sleep 15 # Free API: 4 requests/min done < /workspace/iocs/sha256.txt # MalwareBazaar lookup (free, no rate limit issues) while IFS= read -r hash; do curl -s -X POST "https://mb-api.abuse.ch/api/v1/" \ -d "query=get_info&hash=$hash" \ -o "/workspace/iocs/mb_${hash}.json" done < /workspace/iocs/sha256.txt # AbuseIPDB bulk check ABUSE_KEY="<API_KEY>" while IFS= read -r ip; do result=$(curl -s "https://api.abuseipdb.com/api/v2/check?ipAddress=$ip" \ -H "Key: $ABUSE_KEY" -H "Accept: application/json") score=$(echo "$result" | python3 -c "import sys,json; print(json.load(sys.stdin)['data']['abuseConfidenceScore'])" 2>/dev/null) echo "$ip abuse_score=$score" >> /workspace/iocs/abuse_results.txt done < /workspace/iocs/ipv4_public.txt ``` ## 5. IOC Deduplication and Normalization ```bash python3 << 'PYEOF' import json, hashlib, re from collections import defaultdict with open('/workspace/iocs/extracted.json') as f: iocs = json.load(f) deduped = {} stats = {} for category, items in iocs.items(): # Normalize: lowercase domains/URLs/emails, uppercase hashes if category in ('md5s', 'sha1s', 'sha256s', 'sha512s'): normalized = sorted(set(h.lower() for h in items)) elif category in ('domains', 'urls', 'email_addresses'): normalized = sorted(set(i.lower().rstrip('.') for i in items)) elif category in ('ipv4s', 'ipv6s'): normalized = sorted(set(items)) else: normalized = sorted(set(items)) if normalized: deduped[category] = normalized stats[category] = {'original': len(items), 'deduped': len(normalized)} with open('/workspace/iocs/deduped.json', 'w') as f: json.dump(deduped, f, indent=2) print("Deduplication stats:") for cat, s in sorted(stats.items()): removed = s['original'] - s['deduped'] print(f" {cat}: {s['original']} -> {s['deduped']} (removed {removed} dupes)") PYEOF ``` ## 6. STIX/TAXII Formatting ```bash # Convert extracted IOCs to STIX 2.1 bundle python3 << 'PYEOF' from stix2 import (Bundle, Indicator, Identity, MarkingDefinition, TLP_WHITE, TLP_GREEN, TLP_AMBER) import json, datetime identity = Identity( name="Decepticon TI Team", identity_class="organization" ) with open('/workspace/iocs/deduped.json') as f: iocs = json.load(f)