osint-investigator

# OSINT Investigator Open-source intelligence (OSINT) gathering and analysis pipeline using ARGUS BlackArch tools. Combines theHarvester for email/subdomain harvesting, recon-ng for web reconnaissance, Maltego for graphical link analysis, and SpiderFoot for automated data aggregation. Enriched with aynops for domain recon and CVE-MCP for threat intelligence context. Runs on ARGUS infrastructure with 11 installed BlackArch recon/OSINT tools. ## Prerequisites - **ARGUS host** with BlackArch tools installed - **theHarvester** 4.11.1+ — email and subdomain harvesting - **recon-ng** 5.1.2+ — web reconnaissance framework - **Maltego** 4.11.3+ — graphical link analysis - **aynops** MCP available on localhost — domain recon + scanning - `curl`, `jq` on PATH ## Infrastructure | Component | Location | Purpose | |-----------|----------|---------| | theHarvester | `/usr/bin/theHarvester` | Email, subdomain, and name harvesting | | recon-ng | `/usr/bin/recon-ng` | Web reconnaissance framework | | Maltego | `/usr/bin/maltego` | Graphical link analysis | | SpiderFoot | `/usr/bin/spiderfoot` | Automated OSINT platform | | aynops | localhost MCP | Domain recon: whois, DNS, ports, SSL | | CVE-MCP | localhost MCP | CVE intelligence enrichment | ## Core Commands ### Quick Domain Recon (theHarvester) Harvest emails, subdomains, IPs, and URLs for a target domain: ```bash TARGET="example.com" OUTDIR="/tmp/osint/$TARGET-$(date +%Y%m%d-%H%M%S)" mkdir -p "$OUTDIR" echo "=== theHarvester: Full Harvest ===" theHarvester -d "$TARGET" -b all -f "$OUTDIR/harvest.html" 2>&1 | \ tee "$OUTDIR/harvest-output.txt" # Extract emails echo "" echo "=== Extracted Emails ===" grep -E '[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}' "$OUTDIR/harvest-output.txt" | \ sort -u > "$OUTDIR/emails.txt" cat "$OUTDIR/emails.txt" # Extract subdomains echo "" echo "=== Extracted Subdomains ===" grep -E '^[*]?[a-zA-Z0-9.-]+\.'"$TARGET" "$OUTDIR/harvest-output.txt" | \ sort -u > "$OUTDIR/subdomains.txt" cat "$OUTDIR/subdomains.txt" # Extract IPs echo "" echo "=== Extracted IPs ===" grep -oE '\b([0-9]{1,3}\.){3}[0-9]{1,3}\b' "$OUTDIR/harvest-output.txt" | \ sort -u > "$OUTDIR/ips.txt" cat "$OUTDIR/ips.txt" ``` ### theHarvester — Targeted Source Selection Use specific data sources for focused harvesting: ```bash TARGET="example.com" # LinkedIn-focused (people, roles) echo "=== LinkedIn Harvest ===" theHarvester -d "$TARGET" -b linkedin 2>&1 # DNS brute-force echo "=== DNS Brute-Force ===" theHarvester -d "$TARGET" -b dnsdumpster,crtsh,certspotter 2>&1 # Search engines echo "=== Search Engine Discovery ===" theHarvester -d "$TARGET" -b google,bing,yahoo,duckduckgo 2>&1 # GitHub/code discovery echo "=== Code Repository Discovery ===" theHarvester -d "$TARGET" -b github-code 2>&1 ``` Available sources: `anubis, baidu, bing, binaryedge, bingapi, bufferoverun, censys, certspotter, crtsh, dnsdumpster, duckduckgo, github-code, google, hackertarget, hunter, intelx, linkedin, linkedin_links, netcraft, omnisint, otx, pentesttools, projectdiscovery, rapiddns, rocketreach, securitytrails, shodan, sslcert, subdomaincenter, threatcrowd, threatminer, urlscan, virustotal, yahoo, zoomeye` ### recon-ng — Automated Reconnaissance Initialize and run a recon-ng workspace: ```bash TARGET="example.com" WORKSPACE="osint-$(date +%Y%m%d)" # Create workspace and run modules recon-ng << RECONEOF workspaces create $WORKSPACE add domains $TARGET # Discover contacts modules load recon/contacts-contacts/mailtester run # Certificate transparency modules load recon/domains-certificates/certspotter run # DNS enumeration modules load recon/domains-hosts/bing_domain_web run modules load recon/domains-hosts/google_site_web run # Whois information modules load recon/domains-contacts/whois_pocs run # Shodan integration modules load recon/hosts-ports/shodan_ip run # Generate report modules load reporting/list set FILENAME /tmp/osint/${WORKSPACE}-contacts.txt set TABLE contacts run modules load reporting/list set FILENAME /tmp/osint/${WORKSPACE}-hosts.txt set TABLE hosts run exit RECONEOF echo "Workspace: $WORKSPACE" echo "Report: /tmp/osint/${WORKSPACE}-*.txt" ``` ### DNS Reconnaissance via aynops Use aynops MCP for comprehensive DNS mapping: ```bash TARGET="example.com" echo "=== AynOps DNS Recon: $TARGET ===" # Whois echo "--- Whois ---" curl -s -X POST "http://localhost:8765/aynops/whois" \ -H "Content-Type: application/json" \ -d "{\"domain\": \"$TARGET\"}" | jq '{ registrar: .registrar, created: .creation_date, expires: .expiration_date, nameservers: .name_servers[:5], org: .registrant_organization }' # DNS Records echo "" echo "--- DNS Records ---" curl -s -X POST "http://localhost:8765/aynops/dns" \ -H "Content-Type: application/json" \ -d "{\"domain\": \"$TARGET\", \"type\": \"all\"}" | jq '{ a: .A[:5], mx: .MX[:3], ns: .NS[:5], txt: .TXT[:3], cname: .CNAME[:5], soa: .SOA }' # SSL Certificate Info echo "" echo "--- SSL Certificate ---" curl -s -X POST "http://localhost:8765/aynops/ssl" \ -H "Content-Type: application/json" \ -d "{\"domain\": \"$TARGET\"}" | jq '{ issuer: .issuer, valid_from: .valid_from, valid_until: .valid_until, sans: .subject_alt_names[:10], fingerprint: .sha256_fingerprint[:16] }' ``` ### SpiderFoot — Automated OSINT Scan Run a comprehensive SpiderFoot scan (passive only by default): ```bash TARGET="example.com" SCAN_NAME="osint-$(date +%Y%m%d)" echo "=== SpiderFoot: Passive Scan ===" spiderfoot -s "$TARGET" -t "$SCAN_NAME" -m all 2>&1 | \ tee "/tmp/osint/spiderfoot-$SCAN_NAME.txt" echo "" echo "=== SpiderFoot: High-Risk Findings ===" grep -E "HIGH|CRITICAL|RISK" "/tmp/osint/spiderfoot-$SCAN_NAME.txt" ``` ### Maltego — Graph Export Export Maltego graph data for programmatic analysis: ```bash TARGET="example.com" GRAPH_DIR="/tmp/osint/maltego-$TARGET-$(date +%Y%m%d)" mkdir -p "$GRAPH_DIR" echo "=== Maltego Export Instructions ===" echo "" echo "1. Launch Maltego: maltego" echo "2. New Graph → select 'Company Stalker' or 'Domain Investigation' machine" echo "3. Input: $TARGET" echo "4. Run machine → wait for transforms to complete" echo "5. Export: File → Export as CSV → $GRAPH_DIR/entities.csv" echo "6. Export: File → Export as CSV (connections) → $GRAPH_DIR/links.csv" echo "" echo "After export, analyze:" # Analysis after manual export if [ -f "$GRAPH_DIR/entities.csv" ]; then echo "" echo "=== Entity Summary ===" echo "Total entities: $(wc -l < "$GRAPH_DIR/entities.csv")" echo "" echo "=== Top Entity Types ===" cut -d',' -f2 "$GRAPH_DIR/entities.csv" | sort | uniq -c | sort -rn | head -10 echo "" echo "=== IP Addresses Discovered ===" grep -E '\b([0-9]{1,3}\.){3}[0-9]{1,3}\b' "$GRAPH_DIR/entities.csv" | sort -u echo "" echo "=== Email Addresses ===" grep -E '[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+' "$GRAPH_DIR/entities.csv" | sort -u echo "" echo "=== Domains ===" grep -E '\.[a-z]{2,}$' "$GRAPH_DIR/entities.csv" | cut -d',' -f1 | sort -u | head -20 fi ``` ### Combined OSINT Report Generate a comprehensive OSINT report from all tools: ```bash TARGET="example.com" REPORT_DIR="$HOME/App/domains/argus/reports/osint" REPORT="$REPORT_DIR/osint-report-$TARGET-$(date +%Y%m%d-%H%M%S).md" TMPDIR="/tmp/osint/$TARGET-$(date +%Y%m%d)" mkdir -p "$REPORT_DIR" "$TMPDIR" echo "# OSINT Report: $TARGET" > "$REPORT" echo "**Date:** $(date)" >> "$REPORT" echo "" >> "$REPORT" # 1. Domain Info (aynops) echo "## 1. Domain Registration" >> "$REPORT" curl -s -X POST "http://localhost:8765/aynops/whois" \ -H "Content-Type: application/json" \ -d "{\"domain\": \"$TARGET\"}" | jq -r ' "| Field | Value |\n|---|---|\n" + "| Registrar | \(.registrar // "N/A") |\n" + "| Created | \(.creation_date // "N/A") |\n" + "| Expires | \(.expiration_date // "N/A") |\n" + "| Nameservers | \(.name_servers[:3] | join(", ") // "N/A") |" ' >> "$REPORT" 2>/dev/null # 2. DNS Records (aynops) echo "" >> "$REPORT" echo "## 2. DNS Records" >> "$REPORT" curl -s -X POST "http://localhost:8765/aynops/dns" \ -H "Content-Type: application/json" \ -d "{\"domain\": \"$TARGET\", \"type\": \"all\"}" | jq -r ' "### A Records\n" + (.A[:10] | map(" - " + .) | join("\n") // " None") + "\n\n### MX Records\n" + (.MX[:5] | map(" - " + .) | join("\n") // " None") + "\n\n### NS Records\n" + (.NS[:5] | map(" - " + .) | join("\n") // " None") + "\n\n### TXT Records\n" + (.TXT[:5] | map(" - " + (. | tostring)[:100]) | join("\n") // " None") ' >> "$REPORT" 2>/dev/null # 3. Subdomain Discovery (theHarvester)