debug-fetcher

# Debug-Fetcher Skill Automated fetch failure handling that: 1. **Queries /memory first** - applies learned strategies before trying defaults 2. **Exhausts all strategies** - direct, playwright, wayback, brave, jina, proxy, UA rotation 3. **Stores successes** - saves working strategies to /memory for future runs 4. **Collaborates with humans** - uses /interview when all automated strategies fail ## Quick Start ```bash # Fetch single URL with failure handling ./run.sh fetch https://example.com # Fetch batch with failure handling ./run.sh fetch-batch urls.txt # Check what was learned about a domain ./run.sh recall example.com # Export all learned strategies ./run.sh export-learnings ``` ## How It Works ``` URL Request │ ▼ ┌──────────────────────────┐ │ 1. Query /memory │ │ "What works for this │ │ domain?" │ └──────────────────────────┘ │ ▼ ┌──────────────────────────┐ │ 2. Try learned strategy │ │ (if exists) │ └──────────────────────────┘ │ ▼ (fail or no learned strategy) ┌──────────────────────────┐ │ 3. Exhaust strategies: │ │ - direct fetch │ │ - playwright │ │ - wayback machine │ │ - brave alternates │ │ - jina reader │ │ - proxy rotation │ │ - user-agent rotation │ └──────────────────────────┘ │ ▼ (all fail) ┌──────────────────────────┐ │ 4. Launch /interview │ │ Ask human for help: │ │ - Credentials? │ │ - Mirror URL? │ │ - Manual download? │ │ - Skip this URL? │ └──────────────────────────┘ │ ▼ ┌──────────────────────────┐ │ 5. Store to /memory │ │ - Successful strategy │ │ - Domain patterns │ │ - Human-provided info │ └──────────────────────────┘ ``` ## Memory Schema Each learned strategy stores: | Field | Description | |-------|-------------| | `domain` | Target domain (e.g., "nytimes.com") | | `path_pattern` | URL path pattern (e.g., "/article/*") | | `successful_strategy` | What worked (e.g., "playwright") | | `headers` | Custom headers that helped | | `timing_ms` | How long the fetch took | | `success_rate` | Historical success rate | | `failure_count` | How many times this domain failed | | `last_used` | Timestamp of last use | | `discovered_at` | When strategy was first learned | ## Commands | Command | Description | |---------|-------------| | `fetch <url>` | Fetch single URL with failure handling | | `fetch-batch <manifest>` | Fetch list of URLs with failure handling | | `recall <domain>` | Show learned strategies for domain | | `export-learnings` | Export all strategies to JSON | ### fetch-batch Options | Option | Default | Description | |--------|---------|-------------| | `--concurrency` | 4 | Max concurrent fetches | | `--no-memory` | false | Disable memory integration | | `--output` | - | Output JSON file | | `--json-stream` | false | Output NDJSON per URL (streaming progress) | | `--task-monitor/--no-task-monitor` | true | Enable/disable task-monitor integration | ## Task-Monitor Integration debug-fetcher integrates with the centralized task-monitor for live progress tracking: ```bash # Run batch fetch with task-monitor (enabled by default) ./run.sh fetch-batch urls.txt # View progress in task-monitor TUI cd ~/.pi/skills/task-monitor uv run python monitor.py tui --filter debug-fetcher # Or check state file directly cat /path/to/debug-fetcher/debug_fetcher_task_state.json | jq ``` State file schema: ```json { "completed": 50, "total": 100, "progress_pct": 50.0, "strategies": { "direct": {"attempts": 30, "successes": 25, "rate": 0.833}, "playwright": {"attempts": 20, "successes": 18, "rate": 0.9} }, "stats": { "success_count": 45, "failure_count": 5, "success_rate": 0.9, "learned_count": 12 }, "failures": [...], "status": "running" } ``` ## NDJSON Streaming Output For long-running batch jobs, use `--json-stream` to output one JSON object per line: ```bash ./run.sh fetch-batch urls.txt --json-stream | tee results.jsonl # Each line: # {"url": "https://...", "success": true, "winning_strategy": "playwright", "attempts": 2, "timing_ms": 1234} ``` This enables: - Real-time progress monitoring via `tail -f results.jsonl | jq` - Integration with streaming parsers - Resume from partial runs ## Environment Variables | Variable | Description | |----------|-------------| | `DEBUG_FETCHER_MEMORY_SCOPE` | Memory scope for storing strategies (default: "fetcher_strategies") | | `DEBUG_FETCHER_MAX_RETRIES` | Max retries per strategy (default: 2) | | `DEBUG_FETCHER_INTERVIEW_THRESHOLD` | Min failures before triggering interview (default: 3) | ## Integration with Fetcher Debug-fetcher wraps the standard fetcher skill and adds failure handling capabilities. All fetcher environment variables (BRAVE_API_KEY, FETCHER_EMIT_MARKDOWN, etc.) are respected. ## Examples ### Learning from Failures After fetching a batch of URLs, debug-fetcher stores successful strategies: ```bash # Fetch a batch ./run.sh fetch-batch urls.txt --output results.jsonl # View what was learned ./run.sh recall attack.mitre.org # Output: # Domain: attack.mitre.org # Strategy: playwright # Success rate: 95% # Last used: 2025-01-30 # Next time, playwright will be tried first for attack.mitre.org ./run.sh fetch https://attack.mitre.org/techniques/T1059 ``` ### Human-in-the-Loop Interview When all strategies fail, an interview is generated: ```bash # Fetch batch with failures ./run.sh fetch-batch difficult_urls.txt # Interview generated at: /tmp/interview_abc123.json # Run: ./agents/skills/interview/run.sh /tmp/interview_abc123.json # Example interview questions: # - "Failed 5 URLs from nytimes.com. Do you have credentials?" # - "archive.org not working. Try a mirror URL?" ``` ### YouTube URL Handling YouTube URLs are automatically detected and handled via the `/ingest-youtube` skill: ```bash # YouTube URLs use transcript extraction ./run.sh fetch https://www.youtube.com/watch?v=abc123 # Uses: /ingest-youtube skill for transcript extraction # Falls back to other strategies if transcript unavailable ``` ### Batch Analysis After a batch run, analyze patterns: ```python from debug_fetcher.batch_analyzer import analyze_batch, get_failure_summary # Get summary summary = get_failure_summary(results) # { # "total": 1000, # "success": 850, # "failed": 150, # "success_rate": "85.0%", # "top_failing_domains": [ # {"domain": "nytimes.com", "count": 45}, # {"domain": "wsj.com", "count": 30} # ], # "patterns": [ # "All 45 URLs from nytimes.com returned HTTP 403", # "High failure rate: 50% of failures are paywalled sites" # ] # } ``` ## Fetch Failure vs Extraction Failure **Critical distinction**: "Empty content" can mean two different things: | Symptom | Fetch Failure | Extraction Failure | |---------|---------------|-------------------| | HTTP status | Non-200 | 200 | | `downloads/` content | Empty or error page | Full HTML present | | `extracted_text/` | Empty | Empty (BUG) or HTML (BUG) | | Root cause | Network/auth/JS rendering | `evaluate_result_content()` bug | | Fix | Add to SPA_FALLBACK_DOMAINS, proxy, etc. | Fix extraction code | ### Diagnosing the Difference ```bash # Step 1: Check HTTP status jq '.items[0].status' consumer_summary.json # 200 = fetch succeeded, issue is extraction # 403/401/etc = fetch failed, try other strategies # Step 2: Check raw download head -c 500 downloads/*.html # If contains real content → extraction bug # If contains JS shell only → needs Playwright # If contains error page → fetch failed # Step 3: Check extracted text head -c 500 extracted_text/*.txt # If contains "<!DOCTYPE html>" → extraction returned HTML (BUG) # If contains clean text → working correctly # If empty → extraction failed # Step 4: Verify method used jq '.items[0].method' consumer_summary.json # "aiohttp" = direct fetch # "playwright" = JS rendering used ``` ### Real-World Example (Fixed 2026-02-04) **Bug**: `result.text` contained raw HTML instead of extracted text. **Evidence**: ``` downloads/abc.html → <!DOCTYPE html>...(full HTML) extracted_text/abc.txt → <!DOCTYPE html>...(same HTML - BUG!) ``` **Fix**: `evaluate_result_content()` now replaces `result.text` with `assessment.text` (trafilatura output). **Lesson**: Always compare `downloads/` vs `extracted_text/` - they should be different formats. ## Recovery Actions When human provides help via interview: | Action Type | Description | Example | |-------------|-------------|---------| | `credentials` | Login credentials provided | username/password for site | | `mirror` | Alternative URL to try | archive.org mirror | | `manual_file` | Human downloaded file manually | Path to local PDF | | `skip` | URL not needed | "Not critical" | | `retry` | Try again later | Server was down | | `custom_strategy` | Specific approach suggested | "Use proxy" | ## Files ``` .pi/skills/debug-fetcher/ ├── SKILL.md # This file ├── run.sh # Entry point ├── pyproject.toml # Dependencies └── debug_fetcher/ # Python package ├── __init__.py ├── cli.py # CLI commands ├── memory_schema.py # FetchStrategy dataclass ├── memory_bridge.py # Recall/learn from /memory ├── strategy_engine.py # Strategy exhaustion loop ├── batch_analyzer.py # Analyze batch failures ├── interview_generator.py # Generate /interview JSON ├── interview_processor.py # Process interview responses ├── recovery_executor.py # Execute recovery actions └── pdf_bridge.py # Cross-skill integration with debug-pdf ``` ## Companion Skill: debug-pdf `debug-fetcher` and `debug-pdf` work together in the pipeline: ``` URL → debug-fetcher → /fetcher → /extractor → debug-pdf ↓ ↓ fetch fail extraction fail ↓ ↓ retry/recover analyze PDF issues ↓ ↓ /memory /memory ``` **Shared failure patterns:** | Pattern | debug-fetcher | debug-pdf | |---------|---------------|-----------| | `auth_required` | HTTP 401/403 | N/A | | `access_restricted` | HTTP 403 | N/A | | `paywall_detected` | Soft paywall | N/A | | `password_protected` | N/A | Encrypted PDF | | `scanned_no_ocr` | N/A | No text layer | | `archive_org_wrap` | Wayback wrapper | Wayback wrapper | **Cross-skill notifications:** - When debug-fetcher successfully fetches a PDF but detects issues (password protected, scanned), it notifies debug-pdf via agent-inbox - When debug-fetcher fails to fetch a PDF URL, it notifies debug-pdf for tracking ## Related Skills - `/memory` - Stores learned fetch strategies - `/interview` - Human collaboration for unrecoverable URLs - `/ingest-youtube` - YouTube transcript extraction - `/fetcher` - Core URL fetching functionality - `/extractor` - Content extraction from fetched documents - `/debug-pdf` - Companion skill for PDF extraction failures