| name | ingest-client |
| description | Run the ingestion sweep that pulls a new client's raw knowledge into Dossier — public website (Firecrawl), files (Unstructured), and Microsoft 365/SharePoint (Graph delta + Copilot connectors) — normalized to clean, provenance-stamped markdown ready for extraction. Use when onboarding a client or refreshing their sources. |
Ingest a Client
Sweep a client's sources into clean, deduplicated, provenance-stamped markdown — the input the author-okf / extraction-engineer turn into OKF concepts. Ingestion is assembled, not invented; reliability and provenance are the job.
Procedure
- Inventory sources with the client: public site(s), SharePoint/M365, file shares, Confluence/Drive/etc., and any tribal-knowledge docs. Record access and scope.
- Web → Firecrawl:
Map to discover URLs, crawl for exhaustive coverage → clean markdown. (WebFetch for one-off pages.)
- Files → Unstructured: partition PDFs/Office/HTML/email/images into semantic markdown elements.
- Microsoft 365 / SharePoint → Graph delta query for incremental sync (sites, driveItem, listItem) and/or Copilot connectors (ACL-aware). Remember: deletes/ACL changes need a full crawl; Notion needs a custom connector.
- Normalize → consistent clean markdown; deduplicate near-identical content.
- Stamp provenance on everything —
source URI, fetch timestamp, and access/ACL notes — so every downstream OKF atom can carry source/confidence. Provenance is sovereignty.
- Stage the normalized output in the client's silo, ready for
author-okf.
Rules
- Tenant isolation: a client's raw data stays in their silo — never commingled. (DEC: siloed multi-tenancy.)
- Incremental where possible — use delta tokens/change feeds; don't re-ingest the world.
- Idempotent, resumable, rate-limit-aware. Capture failures to traces.
Status / dependencies
The connectors run on the Agent SDK runtime (platform-engineer). Until that's built, this skill orchestrates what's available (Firecrawl MCP / WebFetch for web; manual file drops for Unstructured) and records gaps for the platform-engineer to close.
Output
A staged, normalized, provenance-stamped corpus in the client's silo + a coverage report (what was ingested, what was skipped and why).