datus-plugin-development

# Datus Plugin Development Skill ## Goal Turn documentation the user hands you (an SDK reference, a REST/OpenAPI spec, a CLI man page, a README, a set of function signatures, …) into an installable **Datus plugin** — a Python package discovered through the `datus.plugins` entry-point group that adds a `datus <name>` command plus optional bundled skills, prompt awareness, bash permissions, and tool transformers. **The contract is declarative.** A plugin ships a single manifest file — `datus-plugin.yml` inside the importable package — that names its CLI entry function, its permissions, its skills directory, its system-prompt template, and its config schema. The only Python you write are plain functions. There is no plugin class, no base class, and no `datus` import anywhere. **This skill is design-first.** It NEVER writes plugin code on the first pass. It reads the provided documentation, produces a **design draft** the user can review, and then **stops and waits for explicit confirmation**. Only after the user approves (or amends) the draft does it implement the package. This skill is fully self-contained: the complete plugin contract is inlined in the **Contract Reference** appendix below. Do not depend on external docs or on reading Datus source code — everything needed to design and build a plugin is here. ## Non-negotiable ground rules Every design and every line of generated code must honor these (details in the appendix): - **A plugin never imports `datus.*`** and never depends on `datus` (or any shared SDK) in `pyproject.toml`. Datus is the config broker; the plugin only ships a manifest plus plain functions that Datus calls. - **One distribution ships exactly one plugin.** Each `pyproject.toml` declares a single `datus.plugins` entry point — never bundle several plugins into one distribution. Code shared across plugins goes into a separate **library** distribution (it registers no entry point) that the plugins depend on. - **The entry-point value is the package name** — `<name> = "datus_plugin_<name>"`, a pure name → package mapping. A legacy `pkg.module:Class` value is rejected by `datus plugin install` and `datus plugin pack`. - **`datus-plugin.yml` lives at the package root** (inside the importable package, next to `__init__.py`) and must ship in the wheel. Only `manifest_version: 1` is required; every other key is optional. - **The CLI entry is a function** — `main(argv: list[str], profile: dict) -> int | None`, named by the manifest's `cli` key. `argv` is everything after `datus <name>` with Datus' own `--profile` / `--config` already stripped; `profile` is the resolved, `${VAR}`-expanded `agent.plugins.<name>.<profile>` dict. Return an exit code; `None` means `0`; do not call `sys.exit()` on the success path. - **Reserved names.** The entry-point name may not be `upgrade`, `skill`, or `plugin`, and may not start with `-`. It doubles as both the `datus <name>` command and the `agent.plugins.<name>` config key, so pick one clean identifier. - **Never surface secrets.** Secret config fields are marked `x-secret: true` in the manifest's `config_schema`; Datus strips them (and every field NOT declared in the schema) before the system-prompt template renders, recursing into declared nested objects — so declare every non-secret field the template needs. Secrets in config are always `${ENV_VAR}` references, never literals. If the requested command name is reserved / collides / starts with `-`, flag it in the draft and propose an alternative. ## Input - `$ARGUMENTS` — path or URL to the source documentation, plus (ideally) the desired command name. If the docs location or the command name is missing, ask for it before drafting. - The user may instead paste the docs directly into the conversation. --- ## Phase A — DESIGN DRAFT (always first; then STOP) ### Step A1: Ingest the source documentation Read the user-provided docs. Extract: - **Capabilities** — the concrete operations the plugin could expose (API endpoints, SDK methods, CLI verbs). For each, note inputs, outputs, and whether it is read-only or state-changing. **Inventory them ALL first** — when the source exposes a large surface (an OpenAPI spec, dozens of resource groups), enumerate every group/endpoint before deciding anything; never silently skip parts of the API. - **Authentication & connection** — base URL / host, auth scheme (bearer token, API key, basic, OAuth), region, workspace/tenant, timeouts. These become profile fields in the `config_schema`. - **Dependencies** — which third-party libraries the wrapper needs (`requests`, `httpx`, the vendor SDK, `typer`, …). Never `datus`. **Relevance filter — Datus is a data-analysis / SQL / warehouse / ETL / metrics agent.** These plugins exist to serve that agent, not to mirror a vendor's whole API. After inventorying every capability, tier each by how well it serves data-analysis workflows and pick a **preliminary scope** that leads with the high-relevance operations. The discriminating axis is **analysis-relevant vs. management/config**, NOT read-vs-write: - Keep analysis-relevant **writes** — authoring metric/query SQL, triggering an ingestion/ETL run, managing a warehouse connection — gated as `ask`. Do not let a "read-only is safe" bias quietly shrink the scope; include the authoring/ETL writes the agent genuinely needs. - Drop analysis-irrelevant capabilities — feature-flag toggles, role/permission admin, project settings — even when they offer tidy CRUD. You never finalize the scope alone. The draft must present, side by side, the **selection rationale** for what you kept AND the **full list of excluded capabilities** with reasons, so the user makes the final call (Step A2 §5). If the docs are ambiguous or incomplete, collect the open questions — do not guess silently. ### Step A2: Produce the draft Render the draft in the conversation using this exact structure. Keep it concrete — every CLI command must trace back to a specific part of the source docs. ``` ## Datus Plugin Design Draft: <name> - CLI command / config key: `datus <name>` · `agent.plugins.<name>` - Purpose: <one line> - Source docs: <what was provided> - Scope: <K of N inventoried capabilities selected — rationale + exclusions in §5> - CLI routing style: <dict dispatch | argparse | REST wrapper | Typer> - Third-party dependencies: <libs, none = pure stdlib> (never datus) ### 1. Configuration parameters (config_schema) | Field | Type | Required | Secret (`x-secret`) | Default | Source doc ref | Notes | |-------|------|----------|---------------------|---------|----------------|-------| | api_base_url | string | yes | no | — | <spec §> | endpoint root | | token | string | yes | YES -> `${VAR}` | — | <auth §> | bearer token | | ... | | | | | | | - This table becomes the manifest's inline JSON Schema (`config_schema`). - Secret fields are configured as `${ENV_VAR}` references, expanded by Datus. - A `name` key is injected by Datus (= profile name); do not declare it. - Every NON-secret field the system-prompt template will reference MUST appear here — undeclared fields are stripped before rendering. - A group of related options (e.g. an `s3` credentials block) is declared as a nested `type: object` property with its own `properties`; use the dotted path as the table's field name (`s3.secret_access_key`) and mark the secret **leaves** `x-secret`, not the whole block, so the non-secret leaves stay editable in the `/plugins` TUI. ### 2. CLI command list For each subcommand: #### `datus <name> <subcommand> [args...]` - Function: <what it does, in one or two lines> - Analysis value: <the data-analysis / SQL / warehouse / ETL / metrics workflow this serves — if you cannot state one, it probably belongs in §5 exclusions> - Doc ref: <SDK method / endpoint / section it maps to> - Permission: - normal: allow | ask | deny — <pattern, e.g. `list-pets:*`> - auto: allow | ask | deny — <pattern> - rationale: <read-only => allow; state-changing => ask under normal> (Repeat per subcommand. Group read-only vs. state-changing so the permission posture is obvious.) This command list is written twice into the manifest: as `permissions` patterns (the posture) and as the `commands` catalogue (the descriptive tree that powers `!<plugin>` completion — see [Declaring the CLI command catalogue](#declaring-the-cli-command-catalogue)). Command groups become nested `subcommands`; note each command's key arguments so the catalogue can hint them. Permission patterns are namespace-relative — Datus prefixes `datus <name> ` automatically. Syntax: `cmd` exact, `cmd:*` prefix, `cmd:glob` prefix + first-arg glob, `:*` whole namespace. Only `normal` and `auto` profiles are accepted. ### 3. Bundled skills & prompt template - Main skill `<name>` — description + when the agent should invoke it. - Setup skill `<name>-setup` — the profile fields it collects from the user, and that it writes `${VAR}` references for secrets (never literals). - Each skill is ONE `SKILL.md` (no `references/`, no `assets/`): list the templates/scripts to inline in it, and split into another skill rather than another file when the content does not fit. - system_prompt template — the NON-SECRET fields to surface per environment (e.g. base URL, region, env name); all of them must be declared in §1. Plan the `{% else %}` installed-but-unconfigured branch pointing at `<name>-setup`. ### 4. Tool transformers (only if applicable) - If the plugin should rewrite/deny the agent's tool calls (e.g. SQL scoping), list the tool patterns and the transform/deny rule. Otherwise: "none". ### 5. Scope selection & excluded capabilities For any non-trivial API, the reader must be able to audit — and overturn — your scoping. Show both halves: - **Selection rationale** — why the §2 commands were kept: which data-analysis / SQL / warehouse / ETL / metrics workflows they serve. Tie back to the relevance filter (Step A1). - **Excluded / deferred capabilities** — EVERY capability inventoried in Step A1 but left out. Group by resource, one-line reason each (analysis-irrelevant / redundant / low priority / needs clarification). Never silently drop parts of a large API — the user decides the final scope from this list, and may pull any excluded item back in. | Capability / resource group | In scope? | Reason | |-----------------------------|-----------|--------| | metrics — read + warehouse-native SQL authoring | yes | core metric analysis | | ingestions — trigger / backfill | yes | ETL the agent drives (`ask`) | | experiments — read results / pulse | yes | analysis output | | feature gates / segments / holdouts | no — deferred | feature-flag mgmt, not data analysis | | roles / users / project settings | no — deferred | access & project admin, not analysis | | ... (list every remaining group) | | | End with an explicit prompt: "Confirm this scope, or tell me which excluded capabilities to include." ### 6. End-to-end testability - Deterministic user goal: <one bounded task the plugin must complete through `datus -p`>. - Environment fixtures: <minikube / MinIO / Flink Operator / other pinned services, or none>. - Independent oracle: <exact external state, schema, count, checksum, or resource fields to verify without calling the tested plugin>. - Generated artifacts: <workspace-relative output patterns to preserve>. - Support plugins and minimum permission prefixes: <only what the workflow needs>. - Efficiency signals: <expected command sequence plus tool-call / turn / token / unexpected-failure budgets>. - Feasibility: <runnable now | reference workflow pending fixture/oracle>, including prerequisites. This section becomes the input to `$build-datus-plugin-e2e` after implementation. The E2E oracle, not the LLM's narrative, owns the correctness verdict. ### 7. Open questions - <anything the docs did not resolve: auth details, pagination, rate limits, command-name conflicts, or scope calls you are unsure about …> ``` ### Step A3: STOP and wait for confirmation After presenting the draft: - Explicitly ask the user to **confirm, amend, or reject** the draft, to **confirm the scope or pull any excluded capability (§5) back in**, and to answer any open questions. - **Do not create any files, do not write any code, do not scaffold the package.** End the turn here. - Proceed to Phase B **only** after the user gives explicit approval. If the user requests changes, revise the draft and stop again. --- ## Phase B — IMPLEMENTATION (only after explicit confirmation) Use the recipes and semantics from the **Contract Reference** appendix. Implement in this order: 1. **Package layout** — `datus-plugin-<name>/` with `pyproject.toml` and a `datus_plugin_<name>/` package (`__init__.py`, `datus-plugin.yml`, `cli.py`, `prompts/`, `skills/`). See [Package layout & entry point](#package-layout--entry-point). 2. **Entry point** — register under `[project.entry-points."datus.plugins"]` as `<name> = "datus_plugin_<name>"` (the package name — no `:Class`). Dependencies list the plugin's own libs, **never `datus`**. 3. **The manifest** — write `datus_plugin_<name>/datus-plugin.yml` from the approved draft: `manifest_version: 1`, `description`, `cli`, `permissions` (§2 table), `system_prompt`, `skills`, `config_schema` (§1 table), and `commands` (the §2 command list as a nested catalogue — see [Declaring the CLI command catalogue](#declaring-the-cli-command-catalogue)). See [The manifest](#the-manifest). 4. **The CLI function** — `main(argv, profile)` using the routing style from the draft (see [CLI recipes](#cli-recipes)). Read endpoint/credentials from `profile`; map subcommands to operations; return conventional exit codes (`0` ok, `1` runtime, `2` usage, `3` config, `8` missing optional dep). 5. **The prompt template** — `prompts/system.md.j2` with a `{% if profiles %}` configured branch and an `{% else %}` branch pointing at `<name>-setup` ([System-prompt template](#system-prompt-template)). 6. **Bundled skills** — `skills/<name>/SKILL.md` and `skills/<name>-setup/SKILL.md`, each **one self-contained file**: no `references/` or `assets/` siblings, every template inlined as a fenced block (see [Bundling skills](#bundling-skills) and [Bundling a setup skill](#bundling-a-setup-skill)). 7. **Packaging** — ensure the manifest, template, and skill files ship in the wheel (Hatchling packages them by default; with setuptools add `[tool.setuptools.package-data]`). 8. **Tests** — call `main(argv, profile)` with a plain dict (no `agent.yml`, no Datus imports); validate the manifest with `yaml.safe_load` and render the template with plain Jinja2. See [Testing your plugin](#testing-your-plugin). 9. **E2E handoff** — after unit and manifest-contract tests pass, invoke `$build-datus-plugin-e2e` when end-to-end coverage is in scope. Implement the approved §6 deterministic goal through the repository's pytest harness; do not add a standalone test CLI and do not use LLM judgment as the oracle. ### Verify against the constraints checklist Before declaring done, confirm every item in [Constraints checklist](#constraints-checklist). ### End-to-end verification - **CLI dispatch**: `datus plugin install src:./datus-plugin-<name>` (or `pip install -e` into datus' environment) then `datus <name> ...`. If it falls through to the REPL, the entry point is missing or misnamed; if it prints "declares no CLI command", the manifest loaded but has no `cli` key.