eval-generator-code-app

name	eval-generator-code-app
argument-hint	[path to your Code App project] [optional: BRD file or OneDrive URL]
description	Generate a two-layer eval suite (presence checks + static security analysis) for a Power Apps Code App. Code Apps use @microsoft/power-apps SDK with generated service classes in src/generated/services/ — NOT @microsoft/powerapps-component-framework or context.webAPI.*. BRD is optional — if absent, features are derived from a code review of the project. Always produces: evals/manifest.json, evals/presence/<id>.check.ts files, evals/runner/run-evals.ts, evals/runner/presence-runner.ts, evals/runner/security-runner.ts, and evals/dashboard/index.html. Triggers: 'generate code app evals', 'eval my code app', 'generate tests for code app', 'code app eval generator', 'check feature completeness code app'.

Eval Generator — Power Apps Code App

Purpose

Answer: "Is all requested functionality present and correctly structured in this Code App?"

Two eval layers are always generated:

1. Presence — static checks (grep/AST) that assert code artifacts exist in the right places
2. Security — static security analysis grounded in the actual source code, covering 12 security categories

Both are mandatory. Neither alone is sufficient.

---

Code App Stack Assumptions

Power Apps Code Apps are full Single-Page Applications (SPAs) — not PCF components. They use the @microsoft/power-apps client library, not @microsoft/powerapps-component-framework. There is no context.webAPI.*, no context.parameters, and no ComponentFramework type anywhere in a Code App.

Concern	Technology
UI framework	React (TSX), Vue, or plain HTML/JS
Bundler	Vite (vite.config.ts) with @microsoft/power-apps-vite plugin (powerApps())
Platform SDK	@microsoft/power-apps
SDK init	initialize() from @microsoft/power-apps/app — called in a PowerProvider wrapper component
Connector access	Generated service classes in src/generated/services/<ApiName>Service.ts
Dataverse operations	Generated <ApiName>Service.ts — static methods, return IOperationResult<T> with .value
Standard connectors	Generated <ConnectorName>Service.ts — static methods, return IOperationResult<T> with .data
Generated models	src/generated/models/<EntityName>Model.ts — TypeScript types for Dataverse entities
Config	power.config.json — project identity, created by npx power-apps init
State management	TanStack Query (@tanstack/react-query), Zustand, or React state
Routing	React Router (react-router-dom)
Package manager	npm
Deployment	npx power-apps push
Eval runner	tsx (TypeScript execution, no transpile step needed)
Static analysis	glob (file discovery in presence checks)

⚠️ Never reference context.webAPI.*, context.parameters, or ComponentFramework anywhere in generated eval files. These are PCF concepts — they do not exist in Code Apps.

---

Step 0 — Snapshot Previous Results

Run this first, before collecting inputs or writing any files.

1. Check if evals/results/latest.json exists in the project root.
2. If it does exist:
   • Read it.
   • Count files matching evals/results/snapshots/iter-*.json to determine N (next iteration = count + 1).
   • Write a copy to evals/results/snapshots/iter-<N>-<ISO-timestamp>.json.
   • Update evals/results/snapshots/index.json (create if absent, append if present).
3. If it does not exist — skip silently. This is the first invocation.

---

Step 0b — Collect Inputs

Ask the user (via m_ask_user for structured choices) for:

1. Requirements document — OPTIONAL. One of:

   • Local file path (.md, .txt, .docx)
   • OneDrive / SharePoint URL
   • Session plan.md (from context)
   • Paste/describe in chat
   • None / Skip — features will be derived from code review (see Step 1b)

2. Code App project root — absolute path to the project directory.

   • Default: current working directory from context.

3. Eval output mode (optional):

   • scaffold+write — write all eval files into the project (default ✅)
   • describe-only — print what would be generated, no file writes

⚠️ Ask ALL questions upfront in a SINGLE m_ask_user call (where applicable) before doing any file work.

---

Step 1 — Read Requirements Document (if provided)

If no BRD / requirements document was provided, skip to Step 1b.

Local .md or .txt

Use the view tool.

Local .docx

Use the docx skill. Call m_get_skill('docx') first.

OneDrive / SharePoint URL

Use m365_download_file (resolve file ID with m365_search_files first).

Session plan.md

Read from the session plan path provided in context.

---

Step 1b — Code-Review Fallback (No BRD)

Run this step ONLY when no requirements document was provided. Applies on first run AND every subsequent iteration.

1. Use glob on src/ to enumerate all files.
2. Use grep to identify:
   • All React components (src/components/**/*.tsx, src/pages/**/*.tsx) → one feature per component/page
   • Developer-written services (src/services/**/*.ts or src/Services/**/*.ts) → one feature per service method cluster
   • Hooks (src/hooks/**/*.ts) → features if they encapsulate connector calls
   • Generated service imports: from '.*generated/services/' → identifies which connectors are wired up
   • Dataverse patterns: <ApiName>Service\.<MethodName>\( in src/ (not src/generated/)
   • TanStack Query: useQuery|useMutation → async data-fetching features
3. For each discovered logical unit, synthesize a feature entry with id, title, description, acceptance_criteria, priority, and connectors.
4. Set manifest.generatedFrom = "code-review".
5. Continue to Step 2 using these code-derived features.

Re-runs: When invoked again without a BRD, repeat Step 1b to pick up new components or services. Match features by id — preserve presence_status for any existing id.

---

Step 2 — Parse Requirements → Feature Manifest

Extract a structured feature list from the requirements document (Step 1) or use the code-derived list from Step 1b.

For each feature, capture:

• id — slugified identifier (e.g. user-search, dataverse-create-record)
• title — short label
• description — what the feature does
• acceptance_criteria — array of verifiable statements (empty if not specified)
• priority — high / medium / low (default medium)
• connectors — array: dataverse, office365users, none
• layer_presence — true (always)

Track in SQL:

CREATE TABLE IF NOT EXISTS features (
  id TEXT PRIMARY KEY,
  title TEXT,
  description TEXT,
  acceptance_criteria TEXT,
  priority TEXT DEFAULT 'medium',
  connectors TEXT,
  layer_presence INTEGER DEFAULT 1,
  presence_status TEXT DEFAULT 'pending',
  mapped_files TEXT
);

Write evals/manifest.json:

{
  "generated": "<ISO timestamp>",
  "project": "<project root basename>",
  "generatedFrom": "brd | code-review",
  "features": [
    {
      "id": "user-search",
      "title": "User Search",
      "description": "Allow users to search for people using the Office365Users connector",
      "acceptance_criteria": [
        "Search input triggers Office365UsersService call",
        "Results display name, email, and job title",
        "Empty query shows placeholder, not error"
      ],
      "priority": "high",
      "connectors": ["office365users"],
      "layers": { "presence"    
    
  

---

Step 3 — Audit the Code App

3a. Verify Code App structure

Use glob to confirm:

• power.config.json exists (created by npx power-apps init — the primary Code App fingerprint)
• package.json exists with @microsoft/power-apps dependency (NOT @microsoft/powerapps-component-framework)
• vite.config.ts or vite.config.js exists and references @microsoft/power-apps-vite
• src/ directory contains TSX/TS source files
• src/generated/ directory exists (indicates connectors added via npx power-apps add-dataverse-api)

If @microsoft/powerapps-component-framework is found instead of @microsoft/power-apps, stop and warn: "This appears to be a PCF component, not a Code App. This skill is designed for Power Apps Code Apps only."

If power.config.json is absent but @microsoft/power-apps is present, warn: "Code App not yet initialized. Run npx power-apps init first." — but continue evaluation.

3b. Map src/ structure

Use glob on src/ to identify:

Pattern	Role
src/App.tsx or src/main.tsx	Root component / entry point
src/pages/<Name>.tsx	Page-level components (with routing)
src/components/<Name>.tsx	Reusable UI components
src/services/<Name>Service.ts or src/Services/<Name>Service.ts	Developer-written service wrappers over generated services
src/hooks/use<Name>.ts	React hooks
src/utils/<name>.ts	Utility functions
src/generated/services/<ApiName>Service.ts	Auto-generated connector service classes — do not test these directly
src/generated/models/<EntityName>Model.ts	Auto-generated TypeScript types
src/generated/appschemas/dataSourcesInfo.ts	Auto-generated — lists all registered data sources
power.config.json	Connector metadata config

Also check for SDK initialization:

grep: "PowerProvider|initialize\(\)" in src/App.tsx or src/main.tsx

If neither is found, flag it: generate a critical presence check for sdk-init-missing.

3c. Detect connector usage

Use grep — all patterns are Code App-specific, never PCF patterns:

# Generated service imports in developer code (src/ excluding src/generated/)
pattern: "from '.*generated/services/" → connector is wired up and used

# Dataverse operations via generated service static methods
pattern: "<ApiName>Service\.\w+\(" → connector operation call
pattern: "result\.value" → Dataverse result access (IOperationResult.value)

# Standard connector result access
pattern: "result\.data" → connector result (IOperationResult.data)
pattern: "result\.success" → success check — should be in every connector call
pattern: "result\.errorMessage" → error handling

# TanStack Query
pattern: "useQuery|useMutation" → async data fetching

# Office365Users (match actual generated service file methods)
pattern: "Office365UsersService\." → any Office365Users service call

⚠️ Do NOT search for context.webAPI.*, retrieveMultipleRecords, createRecord, updateRecord, or deleteRecord — these are PCF patterns and will never appear in a Code App.

Record matched files in the SQL features table (mapped_files column).

3d. Flag unimplemented features

A feature is Not Implemented if its core connector calls are not found in src/.

Mark presence_status = 'not_found' in SQL. Presence checks will explicitly report passed: false with a clear detail message.

---

Step 4 — Generate Presence Checks

Create evals/presence/<feature-id>.check.ts for each feature.

Presence check template

// evals/presence/<feature-id>.check.ts
// AUTO-GENERATED by eval-generator-code-app
// Feature: <feature title>
// Description: <feature description>

import { readFileSync, existsSync } from 'fs';
import { join, dirname } from 'path';
import { fileURLToPath } from 'url';
import { globSync } from 'glob';

// Anchored to this file's own location — works correctly whether invoked via runner or directly
const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);
const PROJECT_ROOT = join(__dirname, '..', '..');  // evals/presence → evals → project root
const SRC = join(PROJECT_ROOT, 'src');
const GENERATED = join(PROJECT_ROOT, 'src', 'generated');

interface PresenceResult {
  featureId: string;
  checks: Array<{ name: ; : ; ?:  }>;
}

  ():  {
   : [] = [];

  
   serviceFile = (, , );
   serviceExists = (serviceFile);
  checks.({
    : ,
    : serviceExists,
    : serviceExists ?  : ,
  });

  
   srcFiles = (, { :  });
   allSource = srcFiles.( (f, )).();

   importFound = .(allSource);
  checks.({
    : ,
    : importFound,
    : importFound ?  : ,
  });

  
   successCheck = .(allSource);
  checks.({
    : ,
    : successCheck,
    : successCheck ?  : ,
  });

   { : , checks };
}

Code App-specific presence patterns

Feature type	Checks to generate
Dataverse operation	src/generated/services/<ApiName>Service.ts exists AND is imported outside src/generated/
Dataverse result	result.value access AND result.success check in files calling the service
Office365Users	src/generated/services/Office365UsersService.ts exists + imported in src/
SDK initialization	PowerProvider or initialize() found in src/App.tsx or src/main.tsx
Developer service wrapper	*Service.ts in src/services/ or src/Services/ with expected method names
Component mounted	TSX file in src/components/ or src/pages/ matching feature name
Generated model usage	Import from ./generated/models/ in feature files
Error handling	result.success === false or result.errorMessage in files calling generated services
Loading state	isLoading, isPending (TanStack Query), or useState loading boolean
Empty state	Conditional render for empty results (length === 0 or similar)
power.config.json	File exists and contains expected connector dataSourceName or connectionId

⚠️ Never generate presence checks that grep for webAPI.*, context.parameters, or ComponentFramework — these will never match in a Code App.

---

Step 5 — Generate Vitest Unit Tests

⛔ This step is MANDATORY on every invocation. Write real unit tests grounded in actual source code. Use it.todo() stubs only when a feature has no testable logic. Never skip this step — even on re-runs.

Re-run behaviour

If evals/unit/ already exists, merge — add tests for any new features, preserve passing tests for existing ones. Never delete existing passing test files.

Grounding mandate — read source before writing any test

⛔ Tests that reference invented method names or field names compile and run but verify nothing. Every fabricated assertion destroys user trust.

Before writing any .test.ts file, you MUST:

1. filesystem-directory_tree on src/generated/services/ and src/lib/ (or src/types/)
2. Read each service file — extract static method names and IOperationResult<T> return shapes
3. Read src/lib/types.ts (or equivalent) — extract all entity interface field names
4. For component tests, read the relevant src/screens/*.tsx — identify which context values and service calls are used

⛔ If a source file cannot be found, write it.todo() stubs and flag in the delivery summary. Do NOT invent signatures.

Always generate: evals/unit/helpers/setup.ts

// evals/unit/helpers/setup.ts
// Vitest global setup — must exist even if empty

Always generate: evals/unit/helpers/mocks.ts

Code Apps use @microsoft/power-apps generated static service classes — NOT context.webAPI.*.

⚠️ CRITICAL — Vitest 2.x: Never use vi.hoisted(). All mock functions must be plain vi.fn() in the module body.

// evals/unit/helpers/mocks.ts
// Power Apps Code App service mocks for Vitest
import { vi } from 'vitest';
import type { AppData } from '../../../src/lib/AppData';

/** Mock IOperationResult<T[]> for service.getAll() */
export function mockGetAll<T>(data: T[] = []) {
  return vi.fn().mockResolvedValue({ success: true, data, error: null });
}

/** Mock IOperationResult<T> for service.get() / .create() / .update() */
export function mockOperation<T>(value: T | null = null, success = true) {
  return vi.fn().mockResolvedValue({
    success,
    value,
    error: success ? null : { message: 'Mock error' },
  });
}

/** Mock AppData context value for renderWithAppData()
 *
 * ⛔ GROUNDING REQUIRED: Before writing this function, read src/lib/AppData.ts
 * and copy the REAL field names from the AppData interface.
 * The field list below is a TEMPLATE PLACEHOLDER — using invented field names
 * compiles but causes TypeScript errors in actual test files that import AppData.
 * Replace every field with the real ones from the project.
 */
export function createMockAppData(overrides?: Partial<AppData>):  {
  
   {
    : ,
    : ,
    
    
    
    ...overrides,
  }    ;
}

Always generate: evals/unit/helpers/factories.ts

⛔ Grounding required. Read src/generated/models/ and src/lib/types.ts BEFORE writing this file. Replace ALL placeholder comments with real field names from those files.

// evals/unit/helpers/factories.ts
// GROUNDED: field names derived from src/generated/models/ and src/lib/types.ts
// One factory per Dataverse entity used in the project

// Replace <EntityName>, <EntityInterface>, <entity_primary_key>, and field names
// with REAL values read from src/generated/models/<EntityName>Model.ts
export function make<EntityName>(overrides?: Partial<<EntityInterface>>): <EntityInterface> {
  return {
    // Primary key — from model file (e.g. demo1_employeeid)
    '<entity_primary_key>': `mock-${Math.random().toString(36).slice(2)}`,
    // Fields — from interface in src/lib/types.ts or src/generated/models/
    // '<demo1_fieldname>': 'Test Value',
    ...overrides,
  } as <EntityInterface>;
}

Always generate: evals/vitest.config.ts

Create in the evals/ folder (NOT the project root):

⚠️ Path anchoring is critical. Vitest sets root to the config file's directory by default — which would be evals/. That makes all relative paths wrong. You MUST explicitly set root to the project root and use resolve() for setupFiles so paths are unambiguous regardless of how vitest is invoked.

// evals/vitest.config.ts
import { defineConfig } from 'vitest/config';
import react from '@vitejs/plugin-react';
import { resolve, dirname } from 'path';
import { fileURLToPath } from 'url';

const __dirname = dirname(fileURLToPath(import.meta.url));
const PROJECT_ROOT = resolve(__dirname, '..');  // evals/ → project root

export default defineConfig({
  plugins: [react()],
  test: {
    globals: true,
    environment: 'jsdom',
    root: PROJECT_ROOT,
    setupFiles: [resolve(__dirname, 'unit/helpers/setup.ts')],
    include: ['evals/unit/**/*.test.ts?(x)'],
    reporters: [['verbose'], ['json', { outputFile: 'evals/results/vitest-results.json' }]],
    pool: 'threads',
    passWithNoTests: true,
  },
});

Unit test file template (one per feature/module)

⛔ Traceability rule — ID must be copied verbatim from manifest.json. Every it() block name must start with the exact id value from evals/manifest.json for that feature — not a paraphrase, not a logical name, not the feature title slug. The runner uses this prefix to map test results to dashboard rows. If the ID doesn't match exactly, the dashboard shows "⏭ No tests" for that feature even though tests ran.

Before writing any test file: open evals/manifest.json, find the feature's "id" field, and copy it character-for-character into every it() name for that feature.

Add a traceability comment above each block:

• With BRD: // AC: <exact acceptance criterion text>
• Without BRD: // Derived from: <ClassName>.<methodName>() — <what this code path does>

After writing all test files: grep each test file for each manifest feature ID to confirm at least one it() references it. If any manifest feature ID has zero matching it() names, fix it before delivering.

// evals/unit/<FeatureName>.test.tsx
// AUTO-GENERATED by eval-generator-code-app
// Feature: <feature title>

import { describe, it, expect, beforeEach, vi } from 'vitest';

// Mock the service class BEFORE importing anything that uses it
// Replace path and class name with REAL values from src/generated/services/
vi.mock('../../src/generated/services/<ServiceClass>', () => ({
  <ServiceClass>: {
    getAll: vi.fn().mockResolvedValue({ success: true, data: [] }),
    create: vi.fn().mockResolvedValue({ success: true, value: {} }),
    update: vi.fn().mockResolvedValue({ success: true, value: {} }),
  }
}));

import { <ServiceClass> } from '../../src/generated/services/<ServiceClass>';
import { make<EntityName> } from './helpers/factories';

describe('<feature-title> — <ServiceClass>', () => {
  beforeEach(() => { vi.clearAllMocks(); });

  // AC: <criterion> OR Derived from: <ServiceClass>.getAll() returns data array
  (,  () => {
     entity = make<>({  });
    (<><>)..({ : , : [entity] });

     result =  <>.();
    (result.).();
    (result.).();
    (result.[].<real_primary_key_field>).(entity.<real_primary_key_field>);
  });

  
  (,  () => {
     payload = make<>();
     <>.(payload  );
    (<>.).(expect.(payload));
  });

  
  
});

AppData context test (always generate for app-data-bootstrap feature)

// evals/unit/AppData.test.tsx
import { describe, it, expect, vi } from 'vitest';
import { renderHook, waitFor } from '@testing-library/react';
import { AppDataProvider, useAppData } from '../../src/lib/AppData';

// Mock all generated services — use REAL service class names from src/generated/services/
// Replace the 6 service imports with REAL names found in the project
vi.mock('../../src/generated/services/Demo1_employeesService', () => ({
  Demo1_employeesService: { getAll: vi.fn().mockResolvedValue({ success: true, data: [] }) }
}));
// ... repeat vi.mock() for each of the other 5 services

describe('AppData context', () => {
  it('app-data-bootstrap: loads all entities on mount', async () => {
    const wrapper = ({ children }: any) => <AppDataProvider>{children}</AppDataProvider>;
    const { result } = renderHook(() => useAppData(), { wrapper });
    await waitFor(() => (result..).());
    
     { _employeesService } =  ();
    (_employeesService.).();
  });

  
  (,  {
    ( ( ())).();
  });
});

---

Step 6 — Security Analysis

Perform static security analysis on src/ on every eval run. No BRD required — 100% grounded in actual source code. Findings are file- and line-referenced.

Security categories

All checks are Code App-specific — @microsoft/power-apps SDK, generated services, Vite SPA patterns. No PCF patterns.

ID	Category	Severity	What to detect
SEC-001	OData Injection	Critical	OData filter strings built by string concatenation or template literal with user input
SEC-002	XSS	Critical	dangerouslySetInnerHTML or direct .innerHTML = with potentially dynamic content
SEC-003	Hardcoded Secret	Critical	api_key, password, secret, token, client_secret assigned a string literal ≥6 chars
SEC-004	PII in Logs	High	console.log/warn/error referencing email, userId, phone, token, Mail, DisplayName etc.
SEC-005	Error Disclosure	High	Raw e.message, result.errorMessage, or String(e) set directly into UI state
SEC-006	Unsafe External Call	High	fetch( to a URL not on *.dynamics.com, *.microsoft.com, or *.microsoftonline.com
SEC-007	Client-Side Auth Bypass	High	UI renders conditional on role/permission check but generated service write has no server-side guard
SEC-008	PII in Browser Storage	Medium	localStorage/sessionStorage.setItem with PII-like key names
SEC-009	Unvalidated Input to Generated Service	Medium	User input variable names passed directly into generated service method calls
SEC-010	Sensitive Data in State	Low	useState holding password, token, secret, or apiKey
SEC-011

Always generate: evals/runner/security-runner.ts

⚠️ CRITICAL: Copy the code below EXACTLY. Do NOT simplify or restructure the scan patterns.

#!/usr/bin/env tsx
// evals/runner/security-runner.ts
// Static security analysis — outputs SecurityResult JSON to stdout

import { readFileSync } from 'fs';
import { join, dirname } from 'path';
import { fileURLToPath } from 'url';
import { globSync } from 'glob';

const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);
const ROOT = join(__dirname, '..', '..');  // evals/runner → evals → project
const SRC = join(ROOT, 'src');

type Severity = 'critical' | 'high' | 'medium' | 'low';

interface SecurityFinding {
  id: string; category: string; severity: Severity;
  file: string; line: number; snippet: ; : ;
}

  {
  : ;
  : ; : ; : ;
  : [];
}

 : [] = [];
 srcFiles = ();

 () {
   ( file  srcFiles) {
     : ;
     { content = (file, ); }  { ; }
     lines = content.();
    lines.( {
       (pattern.(line) && !(exclude && exclude.(line))) {
        findings.({ id, category, severity,
          : file.( + , ).( + , ),
          : i + , : line.().(, ), detail });
      }
    });
  }
}


(, , ,
  ,
  );


(, , ,
  ,
  );





(, , ,
  ,
  ,
  );




(, , ,
  ,
  ,
  );


(, , ,
  ,
  );


(, , ,
  ,
  ,
  );


(, , ,
  ,
  );




(, , ,
  ,
  );




(, , ,
  ,
  );


(, , ,
  ,
  );


(, , ,
  ,
  );


(, , ,
  ,
  );

(, , ,
  ,
  );

(, , ,
  ,
  );


(, , ,
  ,
  ,
  );

(, , ,
  ,
  ,
  );

(, , ,
  ,
  );


(, , ,
  ,
  );

(, , ,
  ,
  );

  = [,,,,,,,,,,,,,];


 {
   configContent = ((, ), );
   secretPattern = ;
   labelPattern  = ;
   (secretPattern.(configContent) && !labelPattern.(configContent)) {
    findings.({
      : , : , : ,
      : , : ,
      : ,
      : ,
    });
  }
}  {  }



 {
   { execSync } =  ();
   auditOut = (, {
    : , : , : , : ,
  });
   audit = .(auditOut);
   vulns = audit. ?? {};
   ( [pkg, v]  .(vulns  <, >)) {
     :  = (v. ===  || v. === ) ? v. : v. ===  ?  : ;
     (sev ===  || sev === ) {
       via = (v. ?? []).(  x === );
       cve = via.( x. ?? x. ?? ).().() || ;
      findings.({
        : , : , : sev,
        : , : ,
        : ,
        : ,
      });
    }
  }
}  (: ) {
  
   {
     out = auditErr. ?? auditErr.?.[] ?? ;
     (out) {
       audit = .(out);
       vulns = audit. ?? {};
       ( [pkg, v]  .(vulns  <, >)) {
         :  = v. ===  ?  : v. ===  ?  : v. ===  ?  : ;
         (sev ===  || sev === ) {
           via = (v. ?? []).(  x === );
           cve = via.( x. ?? x. ?? ).().() || ;
          findings.({
            : , : , : sev,
            : , : ,
            : ,
            : ,
          });
        }
      }
    }
  }  {  }
}

 hitIds =  (findings.( f.));
 :  = {
  : ,
  : .( !hitIds.(c)).,
  : hitIds.,
  : ,
  findings,
};
process..(.(result));

---

Step 7 — Generate the Eval Runner

⚠️ CRITICAL: Copy the code below EXACTLY.

Create evals/runner/run-evals.ts:

#!/usr/bin/env tsx
// evals/runner/run-evals.ts
// Orchestrates all eval layers and writes evals/results/latest.json

import { execSync } from 'child_process';
import { writeFileSync, mkdirSync, readFileSync, existsSync } from 'fs';
import { join, dirname } from 'path';
import { fileURLToPath } from 'url';

// ESM-safe __dirname
const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);

const ROOT = join(__dirname, '..');         // evals/runner → evals
const PROJECT_ROOT = join(ROOT, '..');      // evals → project root
const RESULTS_DIR = join(ROOT, 'results');

interface LayerResult {
  layer: 'presence' | 'unit' | 'security';
  passed: number; failed: number; skipped: ;
  ?: <{ : ; : ; :  |  | ; ?:  }>;
  ?: [];
}

  {
  : ;
  : [];
  : {
    : ;
    : ;
    : ;
    : [];
    : [];
    : <{ : ; : ; : ; : ; : ; :  }>;
  };
}

(, { :  });

 manifest = .(((, ), ));

 :  = {
  :  ().(),
  : [],
  : {
    : manifest..,
    : ,
    : ,
    : manifest. ?? [],
    : [],
    : [],
  },
};


.();
 {
   presenceOut = (, { : , : , :  });
   :  = .(presenceOut);
  results..(presenceResult);
  .();
}  (e) {
  .(, e);
  results..({ : , : , : , : , : [] });
}





.();
 {
  (, {
    : , : , : , : ,
  });
}  {  }

 vitestPath = (, );
 ((vitestPath)) {
   {
     vitestData = .((vitestPath, ));
     : [] = [];
     ( suite  vitestData. ?? []) {
       ( t  suite. ?? []) {
         match = t..();
         featureId = match?.[] ?? ;
         name = match?.[] ?? t.;
        unitDetails!.({
          featureId,
          name,
          : t. ===  ?  : (t. ===  || t. === ) ?  : ,
          : t.?.() || ,
        });
      }
    }
    results..({
      : ,
      : vitestData. ?? ,
      : vitestData. ?? ,
      : vitestData. ?? ,
      : unitDetails,
    });
    .();
  }  {
    results..({ : , : , : , : , : [] });
  }
}  {
  .();
  results..({ : , : , : , : , : [] });
}


.();
 {
   secOut = (, { : , : , : , :  });
   secResult = .(secOut);
  results..(secResult);
   critHigh = (secResult. ?? []).( f. ===  || f. === ).;
  .();
}  (e) {
  .(, e);
  results..({ : , : , : , : , : [] }  );
}


 allFailed = (results.  [])
  .( l. === )
  .( (l. ?? []).( d. === ));
results.. = allFailed.( );
results.. = ((results.  []).( l. === )?. ?? []);
results.. = (results.  []).( l. === )?. ?? ;
results.. = (results.  []).( l. === )?. ?? ;

 latestPath = (, );
(latestPath, .(results, , ));
.();


 dashboardPath = (, , );
 ((dashboardPath)) {
   html = (dashboardPath, );
   baked = .(results, , );
  html = html.(, );
  (dashboardPath, html);
  .();
}


 totalFailed = results..( sum + l., );
 (totalFailed > ) {
  .();
  process.();
}
.();

Also create evals/runner/presence-runner.ts using the exact template below:

#!/usr/bin/env tsx
// evals/runner/presence-runner.ts
// Runs all presence checks and outputs a LayerResult JSON to stdout.

import { join, dirname } from 'path';
import { fileURLToPath } from 'url';
import { globSync } from 'glob';

// ESM-safe __dirname
const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);
const PRESENCE_DIR = join(__dirname, '..', 'presence');

interface CheckResult {
  featureId: string;
  checks: Array<{ name: string; passed: boolean; detail?: string }>;
}

interface LayerResult {
  layer: 'presence';
  passed: number; failed: number; skipped: number;
  details: Array<{ featureId: string; name: ; :  |  | ; ?:  }>;
}

 checkFiles = ().();
 : [] = [];
 passed = , failed = ;

 ( file  checkFiles) {
   {
     mod =  (file);
     :  = mod.();
     ( c  result.) {
       (c.) {
        passed++;
        details.({ : result., : c., :  });
      }  {
        failed++;
        details.({ : result., : c., : , : c. });
      }
    }
  }  (: ) {
    failed++;
    details.({ : file, : , : , : (e?. ?? e) });
  }
}

 :  = { : , passed, failed, : , details };
process..(.(result));

---

Step 8 — Generate Results Dashboard

⛔ DO NOT write your own dashboard HTML. The dashboard template is a pre-built, tested file. Your job is to READ it, make exactly three substitutions, and WRITE it. Nothing else.

Step 7a — Read the template file

Read the template using the view tool:

path: ~/.copilot/m-skills/eval-generator-code-app/dashboard-template.html

Expand ~ to the user's actual home directory (e.g. C:\Users\<username>). Use PowerShell if needed: (Resolve-Path ~).Path.

Step 7b — Make exactly three substitutions

1. Replace every occurrence of PROJECT_NAME with the project's directory basename (e.g. WorkshopsApp)
2. Replace ITERATION_NUMBER with the current iteration number (1 for first run, increment from snapshot count for re-runs)
3. Replace the FEATURES placeholder line:

     /* GENERATED — replace with actual features: { id, title, priority, note? } */
   

   with the actual feature array built from the manifest, one object per line:

   { id: 'user-search', title: 'User Search', priority: 'high' },
   { id: 'dataverse-read', title: 'Dataverse Read', priority: 'high', note: 'Not yet implemented' },
   

Step 7c — Write the result to evals/dashboard/index.html

Write the substituted content to evals/dashboard/index.html in the project.

Step 7d — Self-verify (THREE checks, all mandatory)

After writing, verify all three:

1. Grep the output file for fetch(. If ANY match is found: delete the file immediately and repeat from 7a.
2. Grep the output file for FEATURES = [ — confirm the next non-whitespace line is NOT /* GENERATED. If the placeholder is still present, the substitution failed — delete the file and repeat from 7a.
3. Grep the output file for PROJECT_NAME — if still present, the project name substitution failed — delete and repeat.

Why this approach

The dashboard opens as a file:// URL. Browsers block ALL network requests (fetch, XMLHttpRequest) on file://. The template uses var DATA baked by the runner at eval time — no runtime loading needed, no server required.

---

Step 9 — Update package.json Scripts

Append to the project's package.json scripts section:

{
  "scripts": {
    "eval": "tsx evals/runner/run-evals.ts",
    "eval:presence": "tsx evals/runner/presence-runner.ts",
    "eval:unit": "vitest run --config evals/vitest.config.ts",
    "eval:security": "tsx evals/runner/security-runner.ts",
    "eval:dashboard": "npx open-cli evals/dashboard/index.html"
  }
}

Also add dev dependencies if not already present:

{
  "devDependencies": {
    "tsx": "^4.11.0",
    "glob": "^11.0.0",
    "vitest": "^2.0.0",
    "@vitejs/plugin-react": "^4.3.0",
    "jsdom": "^24.0.0",
    "@testing-library/react": "^16.0.0",
    "@testing-library/jest-dom": "^6.4.0"
  }
}

After editing package.json, run npm install to apply.

Run the eval suite immediately after scaffolding

⚠️ CRITICAL: After npm install completes, always run the eval suite so the dashboard is populated with real results on first open.

Run in the project root:

npx tsx evals/runner/run-evals.ts

---

Step 10 — Deliver Summary in Chat

After all files are written, present:

Files Written

Path	Purpose
evals/manifest.json	Feature registry
evals/presence/<id>.check.ts	Presence checks (one per feature)
evals/runner/run-evals.ts	Eval orchestrator
evals/runner/presence-runner.ts	Presence runner
evals/runner/security-runner.ts	Security analysis runner
evals/unit/helpers/mocks.ts	Service mock helpers
evals/unit/helpers/factories.ts	Test data factories (grounded in model files)
evals/unit/helpers/setup.ts	Vitest global setup
evals/unit/<FeatureName>.test.tsx	Unit tests (one per feature/module)
evals/vitest.config.ts	Vitest configuration
evals/dashboard/index.html	Results dashboard

Feature Coverage Table

Feature ID	Title	Priority	Presence	Unit	Security
user-search	User Search	High	✅ Generated	✅ Generated	✅ Scanned
dataverse-read	Dataverse Read	High	✅ Generated	⚠️ Stub	✅ Scanned

⚠️ = feature not yet implemented in src/ — tests written as it.todo

Running Evals

npm run eval           # run all layers (presence + unit + security)
npm run eval:presence  # presence checks only
npm run eval:unit      # Vitest unit tests only
npm run eval:security  # security analysis only

---

Mandatory Artifact Checklist — NEVER SKIP

⛔ STOP before delivering the summary in Step 9. Verify every file below was written in this invocation.

#	File	Must exist?
1	evals/manifest.json	ALWAYS
2	evals/presence/<id>.check.ts (one per feature)	ALWAYS
3	evals/unit/helpers/mocks.ts	ALWAYS
4	evals/unit/helpers/factories.ts	ALWAYS
5	evals/unit/helpers/setup.ts	ALWAYS
6	evals/unit/<FeatureName>.test.tsx (one or more)	ALWAYS
7	evals/vitest.config.ts	ALWAYS
8	evals/runner/run-evals.ts	ALWAYS
9	evals/runner/presence-runner.ts	ALWAYS
10	evals/runner/security-runner.ts	ALWAYS
11	evals/dashboard/index.html	ALWAYS — read from dashboard-template.html, NEVER write from scratch

Do not substitute, skip, or defer any item. If src/ is empty, write presence checks that return passed: false with a clear detail message.

---

Non-Negotiable Quality Rules

• ✅ BRD is optional. When absent, features are derived from code review (Step 1b). Eval generation always proceeds.
• ✅ evals/manifest.json is always written — it is the source of truth.
• ✅ evals/dashboard/index.html is ALWAYS written — use the exact template from Step 7. Never skip it.
• ✅ Every feature in the manifest gets a presence check file.
• ✅ Presence checks use ESM fileURLToPath(import.meta.url) anchoring — never process.cwd().
• ✅ After all files are written and npm install completes, ALWAYS run npx tsx evals/runner/run-evals.ts.
• ✅ Presence checks must grep actual patterns from the project — no fabricated method names.
• ❌ Never reference context.webAPI.*, context.parameters, or ComponentFramework in any generated eval file — these are PCF patterns, not Code Apps.
• ❌ Never fabricate file paths or method names. Always glob/grep src/ first.
• ❌ Never read entire large source files — use view_range + targeted grep.
• ✅ evals/unit/ folder with at least one .test.tsx file is ALWAYS created — even if all tests are it.todo() stubs.
• ❌ Never skip unit test generation because evals/ or evals/unit/ already exists. On re-runs, add missing unit test files; on first runs, always create them. "Merge" means ADD, never SKIP.
• ✅ evals/vitest.config.ts is ALWAYS created inside the evals/ folder (never at project root) — with root: PROJECT_ROOT and resolve() for setupFiles.
• ✅ Every it() block name starts with the exact id from evals/manifest.json — copied verbatim, not paraphrased. Verify after writing: grep each test file for each manifest feature ID. Any manifest ID with zero hits = broken dashboard mapping.
• ✅ Every it() block name starts with <feature-id>: — required for dashboard mapping.
• ✅ All test files must have valid TypeScript syntax — no pseudo-code.
• ❌ Never use vi.hoisted() — causes silent failures in Vitest 2.x.

---

Error Handling

Situation	Action
Not a Code App (missing power.config.json and SDK dep)	Warn user, ask to confirm path or continue
No BRD / requirements doc provided	Run Step 1b (code review) — never block or ask again
Requirements doc can't be parsed	Ask user to describe features directly in chat
src/ empty or missing	Generate skeleton manifest + placeholder presence checks — still write all mandatory artifacts
Connector never used in src/	Mark connector features as not_found — presence checks report passed: false
package.json missing tsx/glob	Add dev deps and run npm install
Existing evals/ directory	Unit tests: Always follow Step 5 merge rule — add test files for new features, preserve passing ones. NEVER skip unit test generation because evals/ already exists, even if evals/unit/ is missing entirely. Presence/manifest: Ask user: overwrite / merge / abort

---

Invocation Examples

/eval-generator-code-app
Generate evals for my Code App at C:\projects\my-code-app using requirements at C:\docs\BRD.md
Eval my code app against the current plan.md
Check feature completeness for the code app in my current working directory