pilot-document-processing-setup

# Document Processing Setup Deploy 3 agents that automate document ingestion, data extraction, and search indexing. ## Roles | Role | Hostname | Skills | Purpose | |------|----------|--------|---------| | ingester | `<prefix>-ingester` | pilot-stream-data, pilot-share, pilot-archive | Accepts documents, converts to processable format | | extractor | `<prefix>-extractor` | pilot-task-router, pilot-dataset, pilot-receipt | Extracts structured data — tables, entities, amounts | | indexer | `<prefix>-indexer` | pilot-webhook-bridge, pilot-announce, pilot-metrics | Indexes data for search, publishes to downstream systems | ## Setup Procedure **Step 1:** Ask the user which role this agent should play and what prefix to use. **Step 2:** Install the skills for the chosen role: ```bash # ingester: clawhub install pilot-stream-data pilot-share pilot-archive # extractor: clawhub install pilot-task-router pilot-dataset pilot-receipt # indexer: clawhub install pilot-webhook-bridge pilot-announce pilot-metrics ``` **Step 3:** Set the hostname: ```bash pilotctl --json set-hostname <prefix>-<role> ``` **Step 4:** Write the setup manifest: ```bash mkdir -p ~/.pilot/setups cat > ~/.pilot/setups/document-processing.json << 'MANIFEST' <USE ROLE TEMPLATE BELOW> MANIFEST ``` **Step 5:** Tell the user to initiate handshakes with direct communication peers. ## Manifest Templates Per Role ### ingester ```json {"setup":"document-processing","setup_name":"Document Processing","role":"ingester","role_name":"Document Ingester","hostname":"<prefix>-ingester","description":"Accepts documents (PDF, DOCX, images) via upload or webhook, converts to processable format.","skills":{"pilot-stream-data":"Stream raw document bytes to extractor for processing.","pilot-share":"Share converted document files with extractor.","pilot-archive":"Archive original documents for audit and reprocessing."},"peers":[{"role":"extractor","hostname":"<prefix>-extractor","description":"Receives raw documents for data extraction"}],"data_flows":[{"direction":"send","peer":"<prefix>-extractor","port":1002,"topic":"raw-document","description":"Raw documents in processable format"}],"handshakes_needed":["<prefix>-extractor"]} ``` ### extractor ```json {"setup":"document-processing","setup_name":"Document Processing","role":"extractor","role_name":"Data Extractor","hostname":"<prefix>-extractor","description":"Pulls structured data from documents — tables, key-value pairs, entities, dates, amounts.","skills":{"pilot-task-router":"Route documents to specialized extractors by type (invoice, contract, form).","pilot-dataset":"Store extraction results and training data for accuracy improvement.","pilot-receipt":"Confirm document receipt and report extraction status."},"peers":[{"role":"ingester","hostname":"<prefix>-ingester","description":"Sends raw documents"},{"role":"indexer","hostname":"<prefix>-indexer","description":"Receives extracted structured data"}],"data_flows":[{"direction":"receive","peer":"<prefix>-ingester","port":1002,"topic":"raw-document","description":"Raw documents in processable format"},{"direction":"send","peer":"<prefix>-indexer","port":1002,"topic":"extracted-data","description":"Extracted structured data as JSON"}],"handshakes_needed":["<prefix>-ingester","<prefix>-indexer"]} ``` ### indexer ```json {"setup":"document-processing","setup_name":"Document Processing","role":"indexer","role_name":"Search Indexer","hostname":"<prefix>-indexer","description":"Indexes extracted data for search, builds document summaries, publishes to downstream systems.","skills":{"pilot-webhook-bridge":"Push index events and summaries to downstream APIs and search engines.","pilot-announce":"Broadcast new document availability to interested subscribers.","pilot-metrics":"Track indexing throughput, search latency, and document counts."},"peers":[{"role":"extractor","hostname":"<prefix>-extractor","description":"Sends extracted structured data"}],"data_flows":[{"direction":"receive","peer":"<prefix>-extractor","port":1002,"topic":"extracted-data","description":"Extracted structured data as JSON"},{"direction":"send","peer":"external","port":443,"topic":"index-notification","description":"Index notifications to downstream systems"}],"handshakes_needed":["<prefix>-extractor"]} ``` ## Data Flows - `ingester -> extractor` : raw-document events (port 1002) - `extractor -> indexer` : extracted-data events (port 1002) - `indexer -> downstream` : index notifications via webhook (port 443) ## Handshakes ```bash # ingester <-> extractor: pilotctl --json handshake <prefix>-extractor "setup: document-processing" pilotctl --json handshake <prefix>-ingester "setup: document-processing" # extractor <-> indexer: pilotctl --json handshake <prefix>-indexer "setup: document-processing" pilotctl --json handshake <prefix>-extractor "setup: document-processing" ``` ## Workflow Example ```bash # On extractor — subscribe to raw documents: pilotctl --json subscribe <prefix>-ingester raw-document # On indexer — subscribe to extracted data: pilotctl --json subscribe <prefix>-extractor extracted-data # On ingester — publish a document: pilotctl --json publish <prefix>-extractor raw-document '{"filename":"invoice-2024-003.pdf","type":"pdf","pages":2}' # On extractor — publish extracted data: pilotctl --json publish <prefix>-indexer extracted-data '{"filename":"invoice-2024-003.pdf","vendor":"Acme Corp","amount":12500.00}' ``` ## Dependencies Requires `pilot-protocol` skill, `pilotctl` binary, `clawhub` binary, and a running daemon.