TeoSlayer Document Processing Setup
Deploy 3 agents that automate document ingestion, data extraction, and search indexing.
Roles
| Role | Hostname | Skills | Purpose |
|---|---|---|---|
| ingester | <prefix>-ingester |
pilot-stream-data, pilot-share, pilot-archive | Accepts documents, converts to processable format |
| extractor | <prefix>-extractor |
pilot-task-router, pilot-dataset, pilot-receipt | Extracts structured data — tables, entities, amounts |
| indexer | <prefix>-indexer |
pilot-webhook-bridge, pilot-announce, pilot-metrics | Indexes data for search, publishes to downstream systems |
Setup Procedure
Step 1: Ask the user which role this agent should play and what prefix to use.
Step 2: Install the skills for the chosen role:
# ingester:
clawhub install pilot-stream-data pilot-share pilot-archive
# extractor:
clawhub install pilot-task-router pilot-dataset pilot-receipt
# indexer:
clawhub install pilot-webhook-bridge pilot-announce pilot-metrics
Step 3: Set the hostname:
pilotctl --json set-hostname <prefix>-<role>
Step 4: Write the setup manifest:
mkdir -p ~/.pilot/setups
cat > ~/.pilot/setups/document-processing.json << 'MANIFEST'
<USE ROLE TEMPLATE BELOW>
MANIFEST
Step 5: Tell the user to initiate handshakes with direct communication peers.
Manifest Templates Per Role
ingester
{"setup":"document-processing","setup_name":"Document Processing","role":"ingester","role_name":"Document Ingester","hostname":"<prefix>-ingester","description":"Accepts documents (PDF, DOCX, images) via upload or webhook, converts to processable format.","skills":{"pilot-stream-data":"Stream raw document bytes to extractor for processing.","pilot-share":"Share converted document files with extractor.","pilot-archive":"Archive original documents for audit and reprocessing."},"peers":[{"role":"extractor","hostname":"<prefix>-extractor","description":"Receives raw documents for data extraction"}],"data_flows":[{"direction":"send","peer":"<prefix>-extractor","port":1002,"topic":"raw-document","description":"Raw documents in processable format"}],"handshakes_needed":["<prefix>-extractor"]}
extractor
{"setup":"document-processing","setup_name":"Document Processing","role":"extractor","role_name":"Data Extractor","hostname":"<prefix>-extractor","description":"Pulls structured data from documents — tables, key-value pairs, entities, dates, amounts.","skills":{"pilot-task-router":"Route documents to specialized extractors by type (invoice, contract, form).","pilot-dataset":"Store extraction results and training data for accuracy improvement.","pilot-receipt":"Confirm document receipt and report extraction status."},"peers":[{"role":"ingester","hostname":"<prefix>-ingester","description":"Sends raw documents"},{"role":"indexer","hostname":"<prefix>-indexer","description":"Receives extracted structured data"}],"data_flows":[{"direction":"receive","peer":"<prefix>-ingester","port":1002,"topic":"raw-document","description":"Raw documents in processable format"},{"direction":"send","peer":"<prefix>-indexer","port":1002,"topic":"extracted-data","description":"Extracted structured data as JSON"}],"handshakes_needed":["<prefix>-ingester","<prefix>-indexer"]}
indexer
{"setup":"document-processing","setup_name":"Document Processing","role":"indexer","role_name":"Search Indexer","hostname":"<prefix>-indexer","description":"Indexes extracted data for search, builds document summaries, publishes to downstream systems.","skills":{"pilot-webhook-bridge":"Push index events and summaries to downstream APIs and search engines.","pilot-announce":"Broadcast new document availability to interested subscribers.","pilot-metrics":"Track indexing throughput, search latency, and document counts."},"peers":[{"role":"extractor","hostname":"<prefix>-extractor","description":"Sends extracted structured data"}],"data_flows":[{"direction":"receive","peer":"<prefix>-extractor","port":1002,"topic":"extracted-data","description":"Extracted structured data as JSON"},{"direction":"send","peer":"external","port":443,"topic":"index-notification","description":"Index notifications to downstream systems"}],"handshakes_needed":["<prefix>-extractor"]}
Data Flows
ingester -> extractor: raw-document events (port 1002)extractor -> indexer: extracted-data events (port 1002)indexer -> downstream: index notifications via webhook (port 443)
Handshakes
# ingester <-> extractor:
pilotctl --json handshake <prefix>-extractor "setup: document-processing"
pilotctl --json handshake <prefix>-ingester "setup: document-processing"
# extractor <-> indexer:
pilotctl --json handshake <prefix>-indexer "setup: document-processing"
pilotctl --json handshake <prefix>-extractor "setup: document-processing"
Workflow Example
# On extractor — subscribe to raw documents:
pilotctl --json subscribe <prefix>-ingester raw-document
# On indexer — subscribe to extracted data:
pilotctl --json subscribe <prefix>-extractor extracted-data
# On ingester — publish a document:
pilotctl --json publish <prefix>-extractor raw-document '{"filename":"invoice-2024-003.pdf","type":"pdf","pages":2}'
# On extractor — publish extracted data:
pilotctl --json publish <prefix>-indexer extracted-data '{"filename":"invoice-2024-003.pdf","vendor":"Acme Corp","amount":12500.00}'
Dependencies
Requires pilot-protocol skill, pilotctl binary, clawhub binary, and a running daemon.