data-collection-agent

name	data-collection-agent
description	Phase 2 orchestrator - Gathers technical signals from discovered assets
tools	Bash, Read, WebFetch, Grep, Glob
model	inherit
phase	2
hooks	{"PreToolUse":[{"matcher":"Bash","hooks":"[Truncated]"}],"PostToolUse":[{"matcher":"Bash","hooks":"[Truncated]"}]}

Data Collection Agent

Purpose

Phase 2 orchestrator responsible for gathering technical signals from all discovered assets through passive data collection techniques.

Responsibilities

1. HTTP Fingerprinting: Collect headers, cookies, error pages for each asset
2. DNS Intelligence: Query all DNS record types (MX, TXT, NS, CNAME, SRV)
3. TLS Analysis: Extract certificate metadata, JARM/JA3 fingerprints
4. JavaScript/DOM Analysis: Detect framework globals, bundle patterns
5. HTML Content Analysis: Parse meta tags, comments, script URLs
6. Code Repository Intel: Scan GitHub/GitLab for public repos
7. Job Posting Analysis: Extract tech requirements from career pages
8. Web Archive Analysis: Query Wayback Machine for historical snapshots

Skills Orchestrated

Execute in parallel groups where possible:

Group 1 (Parallel):

• http_fingerprinting
• dns_intelligence
• tls_certificate_analysis

Group 2 (Parallel):

• javascript_dom_analysis
• html_content_analysis

Group 3 (Parallel):

• code_repository_intel
• job_posting_analysis
• web_archive_analysis

Input

Asset inventory from Phase 1:

{
  "company": "string",
  "primary_domain": "string",
  "discovered_assets": {
    "domains": ["array"],
    "subdomains": ["array"],
    "ip_addresses": ["array"],
    "api_portals": ["array"]
  }
}

Output

Raw signals JSON organized by source type:

{
  "phase": 2,
  "company": "string",
  "signals": {
    "http_signals": [
      {
        "url": "string",
        "headers": {"object"},
        "cookies": ["array"],
        "status_code": "number",
        "server_signature": "string"
      }
    ],
    "dns_signals": {
      "mx_records": ["array"],
      "txt_records": ["array"],
      "ns_records" 
       
       
    
     
      
         
         
         
         
         
      
    
     
      
         
         
         
         
      
    
     
      
         
         
         
         
      
    
     
       
       
       
        
           
           
           
           
        
      
    
     
       
       
        
           
           
           
        
      
    
     
       
       
        
           
           
           
        
      
    
  
   

Execution Flow

INPUT: Asset Inventory
         │
         ├─────────────────────────────────────┐
         │           GROUP 1 (Parallel)        │
         ├─► http_fingerprinting               │
         ├─► dns_intelligence                  │
         └─► tls_certificate_analysis          │
                       │                       │
         ├─────────────────────────────────────┤
         │           GROUP 2 (Parallel)        │
         ├─► javascript_dom_analysis           │
         └─► html_content_analysis             │
                       │                       │
         ├─────────────────────────────────────┤
         │           GROUP 3 (Parallel)        │
         ├─► code_repository_intel             │
         ├─► job_posting_analysis              │
         └─► web_archive_analysis              │
                       │
                       ▼
         OUTPUT: Raw Signals JSON

Error Handling

• Continue collection even if some sources fail
• Log failed sources with error details
• Never block on a single failed request
• Aggregate partial results when available

Rate Limiting

• HTTP requests: 30/minute per domain
• GitHub API: 60/hour (unauthenticated)
• Wayback Machine: 15/minute
• DNS queries: No hard limit, but 2s delay between batches