cli-verify

# CLI Verify Skill Verifies the current state of the DEP CLI project. Two modes: - **Spec mode**: a plan file with acceptance criteria is present in the conversation — extract its numbered criteria, group by module, and run one parallel subagent per module to verify them. - **Standalone mode**: no plan file in context — trace changed files from git, determine all affected modules (including shared-lib fanout), and run one independent subagent per module in parallel. Both modes use the same subagent execution path. The mode only affects what each subagent is asked to check. Verification subagents never modify code. The main context may save selected local validation targets and apply fixes reported by subagents, as described below. ## Copilot execution Use the available subagent tool, with one independent invocation per module; use the parallel tool wrapper when available. Do not select a read-only exploration agent for verification that must exercise API calls or cleanup. If delegation is unavailable, report a blocked gate instead of claiming verification ran. Pass only workspace-relative paths, the selected sandbox, and the action or criterion list. Each subagent loads its own credentials inside a Node process using `.github/docs/verification-catalog.md`. Never pass secret values or tokens in prompts. The snippets below are templates: use ESM imports with `.js` extensions, PowerShell here-strings, and the in-memory `envMap` and `accessToken`; do not substitute credential values into command text. Redact sensitive fields before reporting API output, even where a template requests raw output. **Module reference**: module directory names, menu labels, lab-pack manifest paths, and the sandboxManagement exceptions (including when and how to use reset) are in `.github/instructions/module-registry.instructions.md`. Read it whenever you need to resolve a module name, manifest path, or determine cleanup scope. --- ## Prerequisite: Validate-config Before any verification that involves calling Adobe APIs: 1. Check if `.github/validate-config.json` exists. - **Missing or incomplete**: for each module that has changed actions (in standalone mode) or is referenced by spec criteria, check if a config entry exists. For any missing entry, prompt once: - List files in `envFiles/`; ask which to use for this module - Ask for the sandbox name for this module - Save or update the entry in `.github/validate-config.json` The file uses per-module keys: ```json { "aepFoundations": { "envFile": "envFiles/<file>.json", "sandbox": "<sandbox>" }, "ajoArchFoundations": { "envFile": "envFiles/<file>.json", "sandbox": "<sandbox>" } } ``` This selects verification targets only. A sandbox reset always requires fresh, explicit confirmation of its name, as required by the module registry. 2. Pass each module's selected `envFile` path and sandbox to its subagent, not the contents of the env file. 3. Each subagent loads its env file and gets an access token in memory using the setup in `.github/docs/verification-catalog.md`. Do not print the token. --- ## Modes - **Spec mode**: a plan file with acceptance criteria is present in the conversation — verify its numbered criteria via parallel subagents. - **Standalone mode**: no plan file in context — trace git changes, determine all affected modules, run one subagent per module in parallel. - **Single criterion**: verify one numbered criterion when the user says "checkpoint 2" or "check criterion 3". --- ## Steps ### 1. Identify what to check **If a plan file with acceptance criteria is in context (spec mode)**: 1. Pull the criteria from the plan file. 2. For each criterion, identify which module it relates to by examining which action file or lib it references: - References to `menus/aepFoundations/` or libs used exclusively by that module → `aepFoundations` - References to `menus/ajoArchFoundations/` or libs used exclusively by that module → `ajoArchFoundations` - References to a shared lib or cross-module concern → assign to all modules that import it 3. Group criteria by module. A criterion may appear in multiple module groups if it touches shared code. 4. Proceed to Step 2. **If no spec is in context (standalone mode)**: run call-chain tracing to determine which modules and actions to exercise. #### Call-chain tracing 1. Run `git diff --name-only main...HEAD` to identify changed files. 2. For each changed file, determine affected modules and actions: - **Action file** (`menus/<module>/[^menu].js`): module = the directory name (e.g. `aepFoundations`); affected actions = [this file]. - **Lib file** (`lib/**/*.js`): grep ALL `menus/` subdirectories for any file that imports it. Collect the unique set of modules found (may be both `aepFoundations` AND `ajoArchFoundations`). Affected actions in each module = every non-destructive action file in that module that imports the lib (directly or transitively via another lib). - **Menu file** (`menus/<module>/menu.js`): module = the directory name; affected actions = every non-destructive action in that submenu. - **YAML/JSON config** (`industry/**`): grep menu files for the file path; collect modules found; affected actions = menu actions that reference the config. 3. Union all affected modules across all changed files. Even if a shared lib change only touched one function, every module that imports that lib must be included — a broken lib breaks all callers. 4. For each module in the union, collect its full affected action list. Skip any action whose file contains `askConfirmDestructive` — mark it `— SKIP (destructive)`. Skip `sandboxManagement` entirely. 5. For each non-skipped action, record it for subagent execution. Proceed to Step 2. --- ### 2. Execute the verification Both spec mode and standalone mode execute through parallel subagents — verification never runs in the main context. #### Print test assignment table Print this table in the main context before spawning anything: ``` ## Test assignments | Module | Subagent | Actions / criteria assigned | | -------------------- | -------- | --------------------------------------- | | AEP Foundations | SA-1 | [comma-separated list] | | AJO Arch Foundations | SA-2 | [comma-separated list] | ``` This table is the only execution state tracked in the main context window. Do not accumulate subagent execution detail here — subagents own that. #### Spawn subagents Spawn **exactly one subagent per top-level module** in a single message (parallel, not sequential). Never combine modules into a single subagent. --- **Standalone mode — subagent prompt template:** ``` You are running a CLI validation check for the [MODULE_LABEL] module of the DEP CLI project in the current workspace. Sandbox: [SANDBOX_NAME] Env file path: [ENV_FILE] Load credentials and acquire a token within your Node process; never print them. Actions to validate: [ACTION_NAME] | [MODULE_LABEL] → [action label in menu] ... For each action: 1. Static wiring check — confirm the action is correctly exported: node --input-type=module --eval "import * as actionModule from './menus/[MODULE]/[action].js'; console.log(typeof actionModule.[FUNCTION_NAME]);" PASS if output is "function". FAIL if "undefined" or error. 2. Import resolution check — confirm no broken imports: node --input-type=module --eval "import './menus/[MODULE]/[action].js';" 2>&1 PASS if silent (exit 0). FAIL if module-not-found or syntax error. 3. Direct lib function calls — call each lib function the action uses, in the order the action calls them, with real credentials. Read the action file first to identify the lib imports and call sequence, then exercise each one: node --input-type=module --eval " import { libFn } from './lib/path/to/lib.js'; const envMap = { API_KEY: '[API_KEY]', IMS_ORG: '[IMS_ORG]', SANDBOX_NAME: '[SANDBOX_NAME]', PREFIX_NAME: 'dep' }; const result = await libFn('[TOKEN]', envMap, /* any required args */); console.log(JSON.stringify(result)); " PASS if result is non-null/non-empty and matches expected shape. Show raw output. FAIL if the call throws, exits non-zero, or returns an unexpected shape. Do NOT call the action function itself — it contains interactive prompts that hang headlessly. Do NOT use curl — direct lib function calls are the authoritative test and catch JS-level errors (missing try/catch, wrong import paths, incorrect API response handling) that curl cannot. Do NOT skip this step — it is the primary way to detect runtime bugs before the user sees them. Live action run: NOT performed — action functions require TTY due to @inquirer/prompts v13. The lib function calls above are the automated equivalent and cover the same API surface. 4. Data-load outcome check — ONLY when the action being verified is `loadProfileData`, `loadRelationalData`, or `deployRelationalFull`. Runs after step 3. Discover industry vertical first: node -e "const fs = require('fs'); const dirs = fs.readdirSync('./industry').filter(d => !d.startsWith('_') && !d.startsWith('.') && d !== 'sources'); console.log(dirs[0]);" a. Flows enabled — call listFlows via lib (ALL three actions): node --input-type=module --eval " import { listFlows } from './lib/flows/listFlows.js'; const envMap = { API_KEY: '[API_KEY]', IMS_ORG: '[IMS_ORG]', SANDBOX_NAME: '[SANDBOX_NAME]', PREFIX_NAME: 'dep' }; const flows = await listFlows('[TOKEN]', envMap); const notEnabled = flows.filter(f => f.state !== 'enabled'); console.log(JSON.stringify({ total: flows.length, enabled: flows.length - notEnabled.length, notEnabled: notEnabled.map(f => ({ name: f.name, state: f.state })) })); " PASS: total > 0 and all flows have state 'enabled' SKIP: total === 0 (nothing deployed — sandbox empty) FAIL: any flow has a state other than 'enabled' b. Base connection — call listConnections via lib (ALL three actions): node --input-type=module --eval " import { listConnections } from './lib/flows/listConnections.js'; const envMap = { API_KEY: '[API_KEY]', IMS_ORG: '[IMS_ORG]', SANDBOX_NAME: '[SANDBOX_NAME]', PREFIX_NAME: 'dep' }; const conns = await listConnections('[TOKEN]', envMap); console.log(JSON.stringify(conns.map(c => c.name))); " PASS: at least one connection returned SKIP: empty array (consistent with sub-check a SKIP) c. Profile / lookup data — ONLY for `loadProfileData`. SKIP this sub-check entirely for `loadRelationalData` and `deployRelationalFull` — relational dataflows write to relational datasets, not the Profile store, so checkProfileHealth does not apply. node --input-type=module --eval " import { checkProfileHealth } from './lib/profileService/checkProfileHealth.js'; import path from 'path'; const envMap = { API_KEY: '[API_KEY]', IMS_ORG: '[IMS_ORG]', SANDBOX_NAME: '[SANDBOX_NAME]', PREFIX_NAME: 'dep' }; const healthYaml = path.resolve(process.cwd(), 'industry', '[VERTICAL]', 'lab-packs', '[LAB_PACK]', 'health.yaml'); const { passed, results } = await checkProfileHealth('[TOKEN]', envMap, healthYaml); console.log(JSON.stringify({ passed, results })); " Lab-pack mapping: loadProfileData → lab-packs/aep-foundations/health.yaml (or ajo-foundations/health.yaml for ajoArchFoundations) PASS: passed === true (all identity expected counts match) SKIP: all identities return 0 events (sandbox empty — nothing deployed yet) FAIL: any identity count is non-zero but does not match `identity.expected`; result includes `children` with per-dataset event breakdown (dataset name — event count) If all applicable sub-checks are SKIP: mark the entire step as — SKIP and note "No deployed data found — run the data load menu option, wait ~15 min for Profile store propagation, then re-run cli-verify." Do NOT mark as FAIL when the sandbox is simply empty — that is a SKIP. Report format for each action: ### [action name] ✓ PASS (or ✗ FAIL or — SKIP or ~ MANUAL) Path: [MODULE_LABEL] → [action label in menu] Wiring: export ✓/✗ imports ✓/✗ Lib calls: [PASS/FAIL + raw output excerpt for each lib function tested] [On FAIL only] Likely cause: ... / Fix direction: ... End with: ## [MODULE_LABEL] summary ✓ N passed ✗ N failed — N skipped ~ N manual TTY note: @inquirer/prompts v13 requires a real TTY. Action functions (in menus/) cannot be called headlessly — they contain interactive prompts. Lib functions (in lib/) have no prompts and CAN be called via `node --input-type=module` without TTY — always do this. Mark ~ MANUAL only when the criterion requires user-observable UI behavior (menu labels, prompt text, keyboard flow) that cannot be verified programmatically. Never mark API behavior as ~ MANUAL. ``` --- **Spec mode — subagent prompt template:** ``` You are running a CLI validation check for the [MODULE_LABEL] module of the DEP CLI project in the current workspace. Sandbox: [SANDBOX_NAME] Env file path: [ENV_FILE] Load credentials and acquire a token within your Node process; never print them. Criteria to verify: [N] [criterion text] ... For each criterion: 1. Determine the verification method: - static — parse YAML/JSON, validate a reference, confirm a file exists - wiring — verify function export and import resolution via node - lib-call — call a lib function directly via node --input-type=module - manual — criterion requires user-observable UI behavior (menu rendering, prompt text, keyboard flow) that genuinely cannot be verified programmatically → mark ~ MANUAL NOTE: "needs an API call" does NOT make a criterion manual — use lib-call instead. 2. Run the appropriate check: Static file existence: node -e "const fs = require('fs'); console.log(fs.existsSync('path') ? 'EXISTS' : 'MISSING')" Static YAML parse: node -e "const y = require('js-yaml'); const fs = require('fs'); console.log(JSON.stringify(y.load(fs.readFileSync('path','utf8')), null, 2))" Wiring — confirm export: node --input-type=module --eval "import * as actionModule from './menus/[MODULE]/[action].js'; console.log(typeof actionModule.[FUNCTION_NAME]);" PASS if "function". FAIL if "undefined" or error. Wiring — confirm imports resolve: node --input-type=module --eval "import './menus/[MODULE]/[action].js';" 2>&1 PASS if silent. FAIL if module-not-found or syntax error. Direct lib function call — preferred for any criterion about API behavior: Read the relevant action file to find which lib functions it calls. Call each one directly: node --input-type=module --eval " import { libFn } from './lib/path/to/lib.js'; const envMap = { API_KEY: '[API_KEY]', IMS_ORG: '[IMS_ORG]', SANDBOX_NAME: '[SANDBOX_NAME]', PREFIX_NAME: 'dep' }; const result = await libFn('[TOKEN]', envMap, /* args */); console.log(JSON.stringify(result)); " PASS if result matches expected shape. FAIL if throws or returns unexpected value. Show raw output — never summarize. Do NOT use curl for API verification — lib function calls catch JS-level errors that curl cannot. Do NOT call action functions (menus/) headlessly — they hang due to @inquirer/prompts v13. 3. Report format for each criterion: ### Criterion [N] — [label] ✓ PASS (or ✗ FAIL or — SKIP or ~ MANUAL) Method: static | wiring | lib-call | manual Sandbox: [sandbox used] Response: [relevant excerpt — always show raw detail] [On FAIL only] Likely cause: [specific file, line, or field] Fix direction: [one sentence — no code] End with: ## [MODULE_LABEL] summary ✓ N passed ✗ N failed — N skipped ~ N manual TTY note: @inquirer/prompts v13 requires a real TTY. Action functions (in menus/) cannot be called headlessly. Lib functions (in lib/) have no prompts and CAN be called via `node --input-type=module` without TTY — always use lib-call for API verification. Mark ~ MANUAL only when the criterion requires user-observable UI behavior that cannot be verified programmatically. ``` --- **Subagents report back two things only:** 1. Status per action/criterion: PASS / FAIL / SKIP / MANUAL 2. For each FAIL: the specific failure detail and a suggested fix direction **After all subagents complete:** the main agent collects both reports, determines what code changes to make, and applies ALL fixes itself — subagents never modify code. Then re-spawn only the subagents for modules with at least one FAIL — independently, in a single parallel message — against the same original action/criteria list. Repeat until all are PASS / SKIP / MANUAL. --- ### 3. Sandbox cleanup Runs **once**, after all verification subagents have completed and the overall verify summary has been printed. Never runs between individual action checks. Runs regardless of pass/fail outcome — a clean sandbox is required before the next test run. #### Discover industry (main context) ```