eval-generator-code-app

# Eval Generator — Power Apps Code App ## Purpose Answer: **"Is all requested functionality present and correctly structured in this Code App?"** Two eval layers are always generated: 1. **Presence** — static checks (grep/AST) that assert code artifacts exist in the right places 2. **Security** — static security analysis grounded in the actual source code, covering 12 security categories Both are mandatory. Neither alone is sufficient. --- ## Code App Stack Assumptions Power Apps Code Apps are **full Single-Page Applications (SPAs)** — not PCF components. They use the `@microsoft/power-apps` client library, not `@microsoft/powerapps-component-framework`. There is no `context.webAPI.*`, no `context.parameters`, and no `ComponentFramework` type anywhere in a Code App. | Concern | Technology | |---|---| | UI framework | React (TSX), Vue, or plain HTML/JS | | Bundler | Vite (`vite.config.ts`) with `@microsoft/power-apps-vite` plugin (`powerApps()`) | | Platform SDK | `@microsoft/power-apps` | | SDK init | `initialize()` from `@microsoft/power-apps/app` — called in a `PowerProvider` wrapper component | | Connector access | **Generated service classes** in `src/generated/services/<ApiName>Service.ts` | | Dataverse operations | Generated `<ApiName>Service.ts` — static methods, return `IOperationResult<T>` with `.value` | | Standard connectors | Generated `<ConnectorName>Service.ts` — static methods, return `IOperationResult<T>` with `.data` | | Generated models | `src/generated/models/<EntityName>Model.ts` — TypeScript types for Dataverse entities | | Config | `power.config.json` — project identity, created by `npx power-apps init` | | State management | TanStack Query (`@tanstack/react-query`), Zustand, or React state | | Routing | React Router (`react-router-dom`) | | Package manager | npm | | Deployment | `npx power-apps push` | | Eval runner | `tsx` (TypeScript execution, no transpile step needed) | | Static analysis | `glob` (file discovery in presence checks) | > ⚠️ **Never reference `context.webAPI.*`, `context.parameters`, or `ComponentFramework` anywhere in generated eval files. These are PCF concepts — they do not exist in Code Apps.** --- ## Step 0 — Snapshot Previous Results > **Run this first, before collecting inputs or writing any files.** 1. Check if `evals/results/latest.json` exists in the project root. 2. If it **does** exist: - Read it. - Count files matching `evals/results/snapshots/iter-*.json` to determine N (next iteration = count + 1). - Write a copy to `evals/results/snapshots/iter-<N>-<ISO-timestamp>.json`. - Update `evals/results/snapshots/index.json` (create if absent, append if present). 3. If it does **not** exist — skip silently. This is the first invocation. --- ## Step 0b — Collect Inputs Ask the user (via `m_ask_user` for structured choices) for: 1. **Requirements document** — OPTIONAL. One of: - Local file path (`.md`, `.txt`, `.docx`) - OneDrive / SharePoint URL - Session `plan.md` (from context) - Paste/describe in chat - **None / Skip** — features will be derived from code review (see Step 1b) 2. **Code App project root** — absolute path to the project directory. - Default: current working directory from context. 3. **Eval output mode** (optional): - `scaffold+write` — write all eval files into the project (default ✅) - `describe-only` — print what would be generated, no file writes > ⚠️ Ask ALL questions upfront in a SINGLE `m_ask_user` call (where applicable) before doing any file work. --- ## Step 1 — Read Requirements Document (if provided) > **If no BRD / requirements document was provided, skip to Step 1b.** ### Local `.md` or `.txt` Use the `view` tool. ### Local `.docx` Use the `docx` skill. Call `m_get_skill('docx')` first. ### OneDrive / SharePoint URL Use `m365_download_file` (resolve file ID with `m365_search_files` first). ### Session plan.md Read from the session plan path provided in context. --- ## Step 1b — Code-Review Fallback (No BRD) > **Run this step ONLY when no requirements document was provided.** Applies on first run AND every subsequent iteration. 1. Use `glob` on `src/` to enumerate all files. 2. Use `grep` to identify: - All React components (`src/components/**/*.tsx`, `src/pages/**/*.tsx`) → one feature per component/page - Developer-written services (`src/services/**/*.ts` or `src/Services/**/*.ts`) → one feature per service method cluster - Hooks (`src/hooks/**/*.ts`) → features if they encapsulate connector calls - Generated service imports: `from '.*generated/services/'` → identifies which connectors are wired up - Dataverse patterns: `<ApiName>Service\.<MethodName>\(` in `src/` (not `src/generated/`) - TanStack Query: `useQuery|useMutation` → async data-fetching features 3. For each discovered logical unit, synthesize a feature entry with `id`, `title`, `description`, `acceptance_criteria`, `priority`, and `connectors`. 4. Set `manifest.generatedFrom = "code-review"`. 5. Continue to Step 2 using these code-derived features. > **Re-runs:** When invoked again without a BRD, repeat Step 1b to pick up new components or services. Match features by `id` — preserve `presence_status` for any existing `id`. --- ## Step 2 — Parse Requirements → Feature Manifest Extract a structured feature list from the requirements document (Step 1) **or** use the code-derived list from Step 1b. For each feature, capture: - **id** — slugified identifier (e.g. `user-search`, `dataverse-create-record`) - **title** — short label - **description** — what the feature does - **acceptance_criteria** — array of verifiable statements (empty if not specified) - **priority** — `high` / `medium` / `low` (default `medium`) - **connectors** — array: `dataverse`, `office365users`, `none` - **layer_presence** — `true` (always) Track in SQL: ```sql CREATE TABLE IF NOT EXISTS features ( id TEXT PRIMARY KEY, title TEXT, description TEXT, acceptance_criteria TEXT, priority TEXT DEFAULT 'medium', connectors TEXT, layer_presence INTEGER DEFAULT 1, presence_status TEXT DEFAULT 'pending', mapped_files TEXT ); ``` Write `evals/manifest.json`: ```json { "generated": "<ISO timestamp>", "project": "<project root basename>", "generatedFrom": "brd | code-review", "features": [ { "id": "user-search", "title": "User Search", "description": "Allow users to search for people using the Office365Users connector", "acceptance_criteria": [ "Search input triggers Office365UsersService call", "Results display name, email, and job title", "Empty query shows placeholder, not error" ], "priority": "high", "connectors": ["office365users"], "layers": { "presence": true, "unit": true } } ] } ``` --- ## Step 3 — Audit the Code App ### 3a. Verify Code App structure Use `glob` to confirm: - `power.config.json` exists (created by `npx power-apps init` — **the primary Code App fingerprint**) - `package.json` exists with `@microsoft/power-apps` dependency (NOT `@microsoft/powerapps-component-framework`) - `vite.config.ts` or `vite.config.js` exists and references `@microsoft/power-apps-vite` - `src/` directory contains TSX/TS source files - `src/generated/` directory exists (indicates connectors added via `npx power-apps add-dataverse-api`) If `@microsoft/powerapps-component-framework` is found instead of `@microsoft/power-apps`, stop and warn: "This appears to be a PCF component, not a Code App. This skill is designed for Power Apps Code Apps only." If `power.config.json` is absent but `@microsoft/power-apps` is present, warn: "Code App not yet initialized. Run `npx power-apps init` first." — but continue evaluation. ### 3b. Map src/ structure Use `glob` on `src/` to identify: | Pattern | Role | |---|---| | `src/App.tsx` or `src/main.tsx` | Root component / entry point | | `src/pages/<Name>.tsx` | Page-level components (with routing) | | `src/components/<Name>.tsx` | Reusable UI components | | `src/services/<Name>Service.ts` or `src/Services/<Name>Service.ts` | Developer-written service wrappers over generated services | | `src/hooks/use<Name>.ts` | React hooks | | `src/utils/<name>.ts` | Utility functions | | `src/generated/services/<ApiName>Service.ts` | **Auto-generated** connector service classes — do not test these directly | | `src/generated/models/<EntityName>Model.ts` | **Auto-generated** TypeScript types | | `src/generated/appschemas/dataSourcesInfo.ts` | **Auto-generated** — lists all registered data sources | | `power.config.json` | Connector metadata config | Also check for SDK initialization: ``` grep: "PowerProvider|initialize\(\)" in src/App.tsx or src/main.tsx ``` If neither is found, flag it: generate a critical presence check for `sdk-init-missing`. ### 3c. Detect connector usage Use `grep` — **all patterns are Code App-specific, never PCF patterns**: ``` # Generated service imports in developer code (src/ excluding src/generated/) pattern: "from '.*generated/services/" → connector is wired up and used # Dataverse operations via generated service static methods pattern: "<ApiName>Service\.\w+\(" → connector operation call pattern: "result\.value" → Dataverse result access (IOperationResult.value) # Standard connector result access pattern: "result\.data" → connector result (IOperationResult.data) pattern: "result\.success" → success check — should be in every connector call pattern: "result\.errorMessage" → error handling # TanStack Query pattern: "useQuery|useMutation" → async data fetching # Office365Users (match actual generated service file methods) pattern: "Office365UsersService\." → any Office365Users service call ``` > ⚠️ Do NOT search for `context.webAPI.*`, `retrieveMultipleRecords`, `createRecord`, `updateRecord`, or `deleteRecord` — these are PCF patterns and will never appear in a Code App. Record matched files in the SQL `features` table (`mapped_files` column). ### 3d. Flag unimplemented features A feature is **Not Implemented** if its core connector calls are not found in `src/`. Mark `presence_status = 'not_found'` in SQL. Presence checks will explicitly report `passed: false` with a clear `detail` message. --- ## Step 4 — Generate Presence Checks Create `evals/presence/<feature-id>.check.ts` for each feature. ### Presence check template ```typescript // evals/presence/<feature-id>.check.ts // AUTO-GENERATED by eval-generator-code-app // Feature: <feature title> // Description: <feature description> import { readFileSync, existsSync } from 'fs'; import { join, dirname } from 'path'; import { fileURLToPath } from 'url'; import { globSync } from 'glob'; // Anchored to this file's own location — works correctly whether invoked via runner or directly const __filename = fileURLToPath(import.meta.url); const __dirname = dirname(__filename); const PROJECT_ROOT = join(__dirname, '..', '..'); // evals/presence → evals → project root const SRC = join(PROJECT_ROOT, 'src'); const GENERATED = join(PROJECT_ROOT, 'src', 'generated'); interface PresenceResult { featureId: string; checks: Array<{ name: string; passed: boolean; detail?: string }>; } export function check(): PresenceResult { const checks: PresenceResult['checks'] = []; // ── Check: Generated service file exists ── const serviceFile = join(GENERATED, 'services', '<ApiName>Service.ts'); const serviceExists = existsSync(serviceFile); checks.push({ name: 'generated service file exists', passed: serviceExists, detail: serviceExists ? undefined : 'Expected: src/generated/services/<ApiName>Service.ts — run npx power-apps add-dataverse-api', }); // ── Check: Service is imported in developer code (outside src/generated/) ── const srcFiles = globSync(`${SRC}/**/*.ts?(x)`, { ignore: `${GENERATED}/**` }); const allSource = srcFiles.map(f => readFileSync(f, 'utf-8')).join('\n'); const importFound = /from ['"].*generated\/services\/<ApiName>Service['"]/.test(allSource); checks.push({ name: 'generated service imported in src/', passed: importFound, detail: importFound ? undefined : 'No import of <ApiName>Service found outside src/generated/', }); // ── Check: result.success is checked (error handling) ── const successCheck = /result\.success/.test(allSource); checks.push({ name: 'result.success checked after service call', passed: successCheck, detail: successCheck ? undefined : 'result.success is never checked — connector errors may be silently swallowed', }); return { featureId: '<feature-id>', checks }; } ``` ### Code App-specific presence patterns | Feature type | Checks to generate | |---|---|