add-dataverse

> **Plugin check**: Run `node "${PLUGIN_ROOT}/scripts/check-version.js"` - if it outputs a message, show it to the user before proceeding. **📋 Shared instructions: [shared-instructions.md](${PLUGIN_ROOT}/shared/shared-instructions.md)** — read first. # Add Dataverse Two paths: - **Existing tables only** — skip to Step 5 (just runs `pa app add data-source` per table) - **New / extended tables** — full workflow with Web API mutations in dependency order ## Workflow 1. Verify project & auth → 2. Resolve plan/operation manifest → 3. Setup Dataverse Web API auth → 4. Validate manifest or reconcile live metadata → 5. Execute sequential metadata phases → 6. Add data sources → 6b. Publish fallback customizations → 6c. Verify tables → 6d. Write manifest → 7. Inspect generated files → 8. Type-check → 8.5. Offline profile reconciliation → 9. Summary --- ### Step 1 — Verify project & auth Confirm Power Apps mobile app: ```bash test -f power.config.json && test -f app.config.js node "${PLUGIN_ROOT}/scripts/resolve-environment.js" "$(node -e \"console.log(require('./power.config.json').environmentId)\")" ``` Capture the **environment URL** (`https://orgXXX.crm.dynamics.com`), **environment ID**, and **tenant ID** from `resolve-environment.js` — needed for Step 3. If only the environment URL is available, pass that URL instead of the ID. ### Step 2 — Resolve plan **Telemetry checkpoint: `resolve_dataverse_schema_plan`** Look for `native-app-plan.md` in the project root: ```bash test -f native-app-plan.md ``` Before reading plan content, inspect `$ARGUMENTS` for the five fast-path artifact flags in Step 2a. When all are present, only confirm `native-app-plan.md` exists for hash validation; do not parse its Data Model section or build operations/service lists from Markdown. **If present and `<operation_manifest_mode> = fallback`:** read the `## Data Model` section. Extract: - The target reconciliation table (`reuse` / `extend` / `create` / `adapt` / `defer` decisions and evidence) - The Mermaid ER diagram (informational) - The "Creation Order" tier list - Every table referenced by `## Screens`, identity resolution, related-entity fields, forms, dashboards, or shared hooks, including standard reused tables such as `systemuser`, `contact`, and `account` Build `SERVICE_REQUIRED_TABLES` as the union of: 1. every non-deferred row in Target Reconciliation (`reuse`, `extend`, `create`, or `adapt`); 2. every table in Creation Order; 3. every table named by screen/hook data requirements. **Hard rule:** `reuse` means "do not mutate schema"; it does **not** mean "skip generated service." If app code reads or writes a reused table, that table must be in `SERVICE_REQUIRED_TABLES`. Carry forward any `adapt` (auto-renamed) and `defer` (out-of-scope this run) decisions with their recorded reasons, and apply the alias map to every name you use. A data-modelling conflict never halts this skill — it resolves to `adapt` or `defer` and is reported in Step 9. **If absent:** check `$ARGUMENTS` for diagram hints (`*.png`, `*.jpg`, `*.jpeg` filename, `erDiagram` keyword, `||--o{` cardinality syntax). - **Diagram hint present** → Path A (Step 2.5). - **No hint AND `$ARGUMENTS` describes what the app does** (the typical case) → silently take Path B (Step 2.6 — spawn architect). No prompt. - **No hint AND `$ARGUMENTS` is empty / non-descriptive** → only then prompt with `AskUserQuestion`: > "How would you like to define the data model? > (a) I have an existing ER diagram to upload (PNG/JPG path, Mermaid syntax, or text description) > (b) Let the data-model-architect agent analyze and propose one (default) > (c) Cancel — I'll plan it elsewhere first" Default the answer to (b) so empty/cancel input auto-proceeds. The 99% case (user gave a description but no diagram) skips this prompt entirely. #### Step 2a — Approved operation-manifest fast path When `$ARGUMENTS` supplies all five paths below, record `<operation_manifest_mode> = candidate`: - `--schema-contract <working_dir>/.tmp/dataverse-schema-contract.json` - `--approval-receipt <working_dir>/.tmp/mobile-plan-status.json` - `--execution-reconciliation <working_dir>/.tmp/dataverse-execution-reconciliation.json` - `--operation-manifest <working_dir>/.tmp/dataverse-operation-manifest.json` - `--publish-checkpoint <working_dir>/.tmp/dataverse-publish-pending.json` Do not reconstruct tables, columns, relationships, keys, payloads, tiers, or service requirements from Markdown on this path. The gate-owned approval receipt binds the exact structured contract content/hash, final plan hash, and final screen/service dependency list; `native-app-plan.md` remains the human review artifact. An entirely absent fast-path handoff means `<operation_manifest_mode> = fallback` and preserves the standalone workflow below, beginning with Step 2 initialization. A partially supplied handoff, or a supplied manifest/contract/reconciliation/checkpoint that is malformed, stale, incomplete, or bound to different context/files, must fail closed: print the exact validation errors and return control to the orchestrator. Never jump to Step 4 without Step 2 initialization, partially trust a candidate, or mix its operations with agent-derived operations. ### Step 2.5 — Path A: Parse user-provided diagram Used when the user has an existing diagram from another tool (Visio, dbdiagram.io, screenshot, hand-drawn). Accept three input formats: | Format | How | |---|---| | **Image path** (`*.png` / `*.jpg` / `*.jpeg`) | Use `Read` on the file path. The vision-capable model extracts entities, columns, relationships. | | **Mermaid syntax** | User pastes a `erDiagram` block in chat. Parse the entities, columns, and `\|\|--o{` cardinalities directly. | | **Text description** | User types a structured description ("Account has many ServiceVisits; each ServiceVisit has many WorkItems and Photos"). Spawn `data-model-architect` agent in `parse-only` mode with the text as input. | Whichever format, normalize into the same structure used by the planner agent: ```yaml publisherPrefix: <from detected publisher prefix or user> tables: - logicalName: contoso_servicevisit displayName: Service Visit status: new # new | extend | reuse columns: [...] relationships: [...] ``` Then: 1. Query existing Dataverse (Step 4 logic) to mark each table as `new`, `modified`, or `reused`. 2. Generate a Mermaid ER diagram from the parsed structure for visual confirmation. 3. Present back to the user via `EnterPlanMode` for approval. 4. On `ExitPlanMode`, write the approved data model into `native-app-plan.md` `## Data Model` section (creating the file if it doesn't exist). 5. Continue to Step 3. ### Step 2.6 — Path B: Spawn architect agent If the user picked Path B (or the user-provided diagram parse failed), spawn the `mobile-app:data-model-architect` agent via `Task` (the `mobile-app:` plugin-name prefix is required) with the user's high-level requirements as input. The agent returns `_dm_section.md`. Embed it in `native-app-plan.md`, present via `EnterPlanMode` for approval, then continue to Step 3. If they need new tables and refuse both paths, recommend they run `/setup-datamodel` (alias of this skill) explicitly, or `native-app-planner` for a full app-level plan. STOP if neither. ### Step 3 — Setup Dataverse Web API auth Required only if creating or extending tables. Skip to Step 5 for read-only `pa app add data-source`. #### Step 3a — Environment consistency check The Power Apps CLI (`$PA`) and `az` authenticate independently — they can point to different accounts. Verify `power.config.json` resolves and `az` can token for the target tenant before making any Dataverse API calls: ```bash ENV_JSON=$(node "${PLUGIN_ROOT}/scripts/resolve-environment.js" "$(node -e \"console.log(require('./power.config.json').environmentId)\")") echo "$ENV_JSON" az account show --query "{user: user.name, tenant: tenantId}" -o json ``` Compare the resolved environment URL with `<envUrl>` captured in Step 1. If they differ, **STOP** and warn: > "⚠️ Environment mismatch detected: > - resolver reports: `<resolved_env_url>` > - This project targets: `<envUrl>` > > The Dataverse API token comes from `az`, which must target the same tenant as the selected environment. Run: > ```bash > az login --tenant <tenant-id> # switch az to the right tenant > ``` > Then re-run `/add-dataverse`." **Do NOT proceed with table creation if environments don't match** — you'll create tables in the wrong org. #### Step 3b — Acquire token ```bash az account show --query "user.name" -o tsv ``` If empty, instruct `az login` and stop. **Script invocation contract — read this once, all subsequent calls in this skill follow it:** ```bash node "${PLUGIN_ROOT}/scripts/dataverse-request.js" <envUrl> <METHOD> <apiPath> \ [--body '<json>'] [--include-headers] \ --tenant-id '<tenantId-from-resolve-environment>' ``` - Three positional args, in order: `<envUrl>`, `<METHOD>` (GET / POST / PATCH / DELETE), `<apiPath>` (everything after `/api/data/v9.2/`). - **Body is a flag, not positional.** `--body '<json>'` — required for POST/PATCH, never for GET/DELETE. Forgetting `--body` and passing the JSON as a 4th positional arg returns a usage error. - `--include-headers` adds response headers (needed for `OData-EntityId` after a record create). - Output is JSON: `{ "status": <code>, "data": <body> }`. Token refresh on 401 and back-off on 429 are automatic — never wrap with manual retry. **Pass the resolved tenant explicitly (HARD — saves discovery and survives fresh shells).** `resolve-environment.js` already returned `tenantId` in Step 1. Substitute that literal value into every `--tenant-id` argument; do not rely on an exported or shell-local variable because separate tool executions may use fresh shells. If the tenant is unknown, omit `--tenant-id` — discovery still works, it is just slower. Acquire a Dataverse access token and verify connectivity: ```bash node "${PLUGIN_ROOT}/scripts/dataverse-request.js" <envUrl> GET WhoAmI \ --tenant-id '<tenantId-from-resolve-environment>' ``` The explicit tenant is also reused for token refresh after a 401 and takes priority over shell environment variables and Azure account discovery. `WhoAmI` is the Dataverse identity endpoint — capital W/A/I (case-sensitive). The response gives `UserId`, `BusinessUnitId`, `OrganizationId` but **does NOT include the publisher prefix**. To get the publisher prefix, query the solution's publisher (defaults to `Default`; pass a different solution name if the env uses a custom solution): ```bash node "${PLUGIN_ROOT}/scripts/detect-publisher-prefix.js" <envUrl> [solutionName] \ --tenant-id '<tenantId-from-resolve-environment>' # solutionName defaults to "Default" if omitted ``` This runs the OData query: `/api/data/v9.2/solutions?$select=uniquename&$expand=publisherid($select=customizationprefix)&$filter=uniquename eq '<solutionName>'` Capture `customizationprefix` from the solution's publisher (typical value: `cr123` → schema names like `cr123_jobsite`). Also capture the solution `uniquename` — needed for the `--solution` flag on every Step 5 / 5b POST so artifacts land in our solution rather than landing wherever Dataverse defaults. Write both to `memory-bank.md` Power Platform context block. Requires the user to hold **System Administrator** or **System Customizer** in this environment. When `<operation_manifest_mode> = candidate`, validate the manifest now against the resolved environment, tenant (when available), publisher, solution, current plan bytes, structured-schema bytes, and fresh reconciliation bytes: ```bash node "${PLUGIN_ROOT}/scripts/build-dataverse-operation-manifest.js" \ --validate "<operation-manifest-path>" \ --contract "<schema-contract-path>" \ --approval-receipt "<approval-receipt-path>" \ --reconciliation "<execution-reconciliation-path>" \ --plan "<working_dir>/native-app-plan.md" \ --environment-id "<environmentId>" \ --env-url "<envUrl>" \ --tenant-id "<tenantId>" \ --publisher-prefix "<customizationprefix>" \ --solution "<solution-uniquename>" \ --publish-checkpoint "<publish-checkpoint-path>" \ --require-executable ``` Validation deterministically rebuilds the expected manifest from the bound structured schema, fresh reconciliation, plan, context, and pending-publish checkpoint, then compares the complete decisions, services, aliases, phases, API paths, and bodies. If validation fails, print every reported mismatch and fail closed to the orchestrator. Do not execute or salvage individual operations and do not switch a supplied candidate to the standalone fallback. Validate with `--require-executable`. Step 8 already performed the one fresh bounded reconciliation for every approved exact table and all of its columns/relationships/keys, including the child/parent/M:N relationship capability managed properties. Missing capability evidence fails closed. If the manifest remains non-executable, report its verification conflicts to the orchestrator. Do not add another read loop, change an approved decision, or enter fallback mode. A non-executable candidate authorizes no metadata write. If validation with `--require-executable` succeeds, set `<operation_manifest_mode> = valid` and continue directly to Step 5's manifest execution branch. This is the fast-v2 path: it skips the repeated agent-driven full reconciliation, not any safety check. ### Step 4 — Reconcile every planned table and column against the target **Telemetry checkpoint: `reconcile_dataverse_schema`** If `<operation_manifest_mode> = valid`, print: > `✓ Approved operation manifest validated — complete fresh reconciliation and derived metadata coverage are bound to this environment.` Use its `decisions` as the reconciliation matrix and skip the remainder of Step 4/4a. Continue to Step 5. A valid manifest has no `unverified` items; its explicit `reuse`, `adapt`, and `defer` rows remain visible in the final summary. **Print before starting:** > "→ Reconciling every planned table and column against live target metadata before any write…" Do not use the custom-table list as the source of truth, and do not issue one request per table. Fetch **every** plan entry (`Reuse`, `Extend`, or `Create`) — including standard and managed dependencies — in a **single** filtered query that also expands their columns: ```bash node "${PLUGIN_ROOT}/scripts/dataverse-request.js" <envUrl> GET \ "EntityDefinitions?\$select=MetadataId,LogicalName,SchemaName,IsCustomEntity,IsManaged,IsCustomizable,CanCreateAttributes,PrimaryIdAttribute,PrimaryNameAttribute&\$filter=LogicalName eq '<table1>' or LogicalName eq '<table2>'&\$expand=Attributes(\$select=MetadataId,LogicalName,AttributeType,AttributeTypeName,RequiredLevel,IsManaged,IsCustomizable,IsPrimaryId,IsPrimaryName,SourceType,SourceTypeMask)" \ --tenant-id '<tenantId-from-resolve-environment>' ``` Build the `$filter` by OR-ing every planned logical name. This is the [documented way to query multiple table definitions at once](https://learn.microsoft.com/power-apps/developer/data-platform/query-schema-definitions#basic-retrievemetadatachanges-example), and it replaces 2N requests (one entity GET plus one attributes GET per table) with one. Keep the `$expand` `$select` list to base `AttributeMetadata` properties only — a single query [cannot cast to a derived column type](https://learn.microsoft.com/power-apps/developer/data-platform/query-schema-definitions#evaluate-other-options-to-retrieve-schema-definitions), so fetch `OptionSet` details separately for the rare column that needs them. Read the results as follows: