fetcher

# Fetcher Fetcher is the project-owned retrieval contract for agents that need public HTTP/HTTPS URLs, local files, PDFs, or explicitly enabled anonymous FTP resources converted into deterministic artifacts. It is not the right tool for OAuth connectors, private SaaS APIs, credentialed scraping, or unbounded recursive crawls. ## First Command Check the installed contract before fetching: ```bash # fetcher-doc-smoke: version-json fetcher version --json ``` Require: - `package.name == "fetcher"`. - `schemas.consumer_summary` includes `fetcher.consumer_summary.v1`. - `entrypoints.fetcher` and `entrypoints.fetcher-etl` are present. - Any optional capability you need is available or explicitly enabled. ## Consumer CLI Use the consumer CLI for agent-facing retrieval and artifact handoff: ```bash fetcher get https://example.com --json --out run/fetcher/example fetcher get-manifest urls.txt --json --out run/fetcher/batch fetcher get-manifest - --json --out run/fetcher/stdin < urls.txt ``` The primary artifact is always: ```text <out>/consumer_summary.json ``` When `--json` is used, stdout is exactly the same summary object. Preserve the process exit status and parse the JSON before using artifacts. Terminal acceptance: - `run_status` is one of `completed`, `completed_with_failures`, `capability_unavailable`, `usage_error`, or `fatal_error`. - `exit_code` matches the process exit code. - `schema == "fetcher.consumer_summary.v1"`. - `items` has one terminal item per requested URL in stable input order. - For synthesis, use `items[].artifacts.extracted_text_path` or `items[].artifacts.markdown_path`, not HTTP status or raw HTML alone. - Treat item `warnings` and `errors` as part of the result, not as log noise. Exit codes: - `0`: every required requested item was accepted. - `2`: usage, manifest, or validation error. - `3`: completed with one or more failed or rejected items. - `4`: requested capability unavailable before it could run. - `5`: fatal internal/orchestration error. ## Tested Smoke Commands These commands are intentionally side-effect-light and are executed by `scripts/ci/fetcher_skill_contract_smoke.py` against a clean wheel: ```bash # fetcher-doc-smoke: doctor fetcher doctor # fetcher-doc-smoke: dry-run-single fetcher get "$FETCHER_SMOKE_URL" --dry-run --json --out "$FETCHER_SMOKE_ROOT/dry-single" # fetcher-doc-smoke: dry-run-manifest fetcher get-manifest "$FETCHER_SMOKE_MANIFEST" --dry-run --json --out "$FETCHER_SMOKE_ROOT/dry-manifest" # fetcher-doc-smoke: ftp-disabled exit=4 fetcher get "ftp://ftp.example.com/pub/data.txt" --json --out "$FETCHER_SMOKE_ROOT/ftp-disabled" # fetcher-doc-smoke: etl-find fetcher-etl --find metrics ``` ## FTP FTP is disabled by default. Enable it only when the caller explicitly needs anonymous read-only `ftp://` retrieval: ```bash fetcher get "ftp://ftp.gnu.org/README" --enable-ftp --json --out run/fetcher/ftp FETCHER_ENABLE_FTP=1 fetcher-etl --manifest ftp-urls.txt --out run/fetcher/ftp-etl ``` Fetcher rejects `ftps://`, SFTP, embedded credentials, authentication, active mode, writes, and recursive directory crawling. Private/local destinations are denied unless `FETCHER_FTP_ALLOW_PRIVATE=1` is set for a trusted fixture. ## ETL Mode Use `fetcher-etl` when you need full pipeline controls, metrics, resolver knobs, inventory JSONL, or ETL audit files: ```bash fetcher-etl --manifest urls.txt --out run/fetcher/etl fetcher-etl --inventory urls.jsonl --output run/fetcher/results.jsonl --audit run/fetcher/audit.json fetcher-etl --help-full fetcher-etl --find metrics ``` Consumer and ETL artifacts are different contracts. Do not assume ETL `results.jsonl` fields are the same shape as `consumer_summary.json`. ## Python API Use Python only when a CLI process is not the right integration boundary. A complete async example must read metadata through `FetchResult.metadata` or `to_dict()`: ```python import asyncio from fetcher.workflows.web_fetch import FetchConfig, URLFetcher async def main() -> None: fetcher = URLFetcher(FetchConfig(concurrency=2, per_domain=1)) results, audit = await fetcher.fetch_many([{"url": "https://example.com"}]) result = results[0] payload = result.to_dict() print(payload["status"]) print((result.metadata or {}).get("content_verdict")) print(audit.get("requested")) asyncio.run(main()) ``` ## Failure Reporting For each degraded or failed item, report: - `requested_url` and `final_downloaded_url`. - `status`, `method`, `content_type`, and `verdict`. - `warnings`, `errors`, `paywall_verdict`, and `alternate_provider`. - Selected artifact paths and whether they exist and are non-empty. - Relevant `failure_summary` buckets such as fallback reason or content verdict. Do not call a run successful from HTTP 200 alone. Use the summary verdict and artifact existence. ## References - `references/USAGE_CONTRACT.md`: artifact selection and acceptance rules. - `references/ETL_AND_CONFIG.md`: ETL-only flags, cache knobs, proxy rotation, alternates, PDF discovery, and Python API details. - `references/TRIGGER_EVAL.md`: should-trigger and should-not-trigger prompts. - `docs/DOWNSTREAM_WRAPPER_CONTRACT.md`: contract for downstream skill wrappers.