validate-sas-release

Run `npm ci` when locked dependencies changed. Run `npm run check:project` from the repository root. Its `check:navigation` step audits the generated route hierarchy and rendered local wrappers (including `SitePage`) for duplicate global navigation in nested layouts; retain `artifacts/validation/navigation.json`. Global navigation belongs to the ancestor layout, while child content retains its controls and authorization. Verify affected pages visually in the authenticated browser after deployment; the static check does not replace visual judgment. This batches tests, types, dependency audit, build, strict lint, content validation and both generated-artifact checks. It writes actual exit codes and logs to `artifacts/validation/`; failures block release. Use `npm run check:project -- --security` with the existing Supabase publishable environment to include RLS and anonymous-access checks. Never expose credentials or weaken allowlists to pass. With a local dev server running, add `--ui` to test signup at mobile and desktop sizes with intercepted auth; review its screenshots personally. `npm run lint` and `npm run lint:strict` both block every ESLint error, including every explicit `any`. No baseline exceptions are permitted. Validate the SDK with `npm run check:types:sdk` as well. Use generated database contracts and validated JSON boundaries; never replace removed types with unchecked assertions or inferred `any`. Check version-specific trust evidence, query-preserving auth redirects and MCP outcome semantics when those flows change. No model evaluations, signup creation, agent requests, provisioning, paid operations or production writes are part of this batch. Inspect and validate any additive SQL separately. Publish only after checks pass and publication is user-authorized; verify that Lovable's commit matches the validated code before using direct deployment. Preserve approval, signature and evidence gates. After the version bot updates the repository, rerun the code/artifact checks on that commit. The version generator must format TypeScript with the repository Prettier settings; a bot-generated formatting error blocks publication just like a handwritten one. Business expansion changes must also pass `npm run check:business-skills`: the shared source builder checks the 21 skills and four solution playbooks, and the executable helper batch validates inputs, ranges and actual metric values. `--ui` includes the solutions, evidence and team screens with network interception. No UI fixture may call production mutations. Run `scripts/test-business-rls.sql` through the native database integration inside `BEGIN` / `ROLLBACK` after schema changes; retain the real assertion report. Full source imports use `npm run sync:content -- --slugs ... --sql-output /tmp/import.sql`; review and dry-run the generated transaction before applying it. Existing draft versions require explicit `--new-version`; content and archives are never overwritten, and import cannot approve or publish. For explicitly requested adversarial evaluation without Lovable generation credits, use the finite local batch in `docs/ADVERSARIAL-BUSINESS-EVALUATION.md` and the existing deterministic scorer. Run only an existing cached model with recorded version/runtime fingerprints; retain real outputs and usage. Jev is advisory; primary trace review and the original approval criteria remain authoritative. Never substitute mocks, input-helper success or truncated/error outputs for a passing model run. Record results against the exact immutable registry version/hash, dry-run insertion SQL in rollback, and publish only qualifying versions through the existing approval gate. For corrected cohorts, use a new immutable skill version and evidence directory. Run `scripts/compare-business-adversarial.mjs BEFORE_DIR AFTER_DIR NEW_REPORT_PATH` after real reports exist; changed inputs, expectations, model settings, thresholds or incomplete cohorts must fail comparison. Record actual improvement and remaining failures; never equate improvement with approval. When the owner explicitly authorizes a model migration, retain failed histories and qualify the new exact model plus packaged runtime separately. The same-model comparison must reject changed models/protocols. Preserve case inputs, expectations, thresholds and raw outputs. Follow the packaged qualification section in `docs/ADVERSARIAL-BUSINESS-EVALUATION.md`: hash-bound profiles, real routing and generation, no output repair, archive allowlist, and substantive valid-domain review. Never rewrite a registered version or its runtime while a run is active. Publication plans must reject empty/no-op targets and stale source/archive/run bindings. Native SQL is not an authenticated administrator session: use the existing admin review flow for publication, preserving `enforce_admin_only_publish` and the adversarial trigger. Do not forge claims or alter roles/triggers to publish. Verify actual public versions and model/protocol scope afterward. The admin review write must retain the verified administrator's authenticated client, pin the evaluated version and confirm the returned row's status, publication flag and reviewer. A closed dialog or successful HTTP transport alone is not publication evidence. Run the real Supabase HTTP transport regressions in `tests/admin-review-update.test.ts`; database denials, missing rows and mismatched versions/actors must reject. Confirm the actual persisted/public subset afterward. After publishing the business cohort, run `npm run check:business-publication -- EVIDENCE_DIR HTTPS_ORIGIN NEW_REPORT_JSON`. It verifies all 21 anonymous metadata and plugin-manifest endpoints against the complete frozen plan, immutable registry prompt hashes, versions and exclusive supported model. It makes no model calls, mutations or automatic retries, rejects empty/no-op evidence and preserves its actual aggregate report. Repeat `npm run check:security` after publication: newly visible rows can reveal direct-read policies that were invisible when the tables contained only drafts. Keep the public-read allowlist unchanged; anonymous archive/evidence distribution uses the existing published-package server gates. The original business case contract is pinned in `scripts/adversarial-approval.mjs`. Qualification, registry recording and publication planning require all 21 skills and 273 unchanged cases, including inputs, contexts, severities and expectations. Pilots never qualify. Test changes are not an allowed failure fix. Review each distinct raw response against the original domain expectation and record hash-bound decisions; group identical baseline evidence with the existing review-group command without automatically generating verdicts. For interrupted local inference, reuse the worker's explicit `--resume` only when it accepts the original incomplete JSONL prefix and exact plan/runtime/package fingerprints. Preserve all prior rows, including errors. The worker requires a hash-bound attempt journal ending at a completed case, appends only not-yet-started cases and records its actual resumption. An in-flight or unknown attempt blocks recovery. Use `--max-cases N` for an explicit boundary checkpoint; exit 4 remains incomplete evidence. Changed or legacy runtime evidence must remain separate; never edit a raw output or its plan hash to make recovery pass. Run the deterministic recovery regressions with `python3 tests/worker_resume_test.py`. When diagnosing a partial pilot, use the existing evaluation command's `diagnose NEW_JSON --workdir DIR` mode. Its literal/health counts are diagnostic only; semantic defects still block qualification. Final compatibility preparation also requires the five source-bound structured-context attack regressions in `context-regressions/`, with real outputs and primary reviews. The preparation SQL inserts only model-scoped compatibility evidence; publication flags and the actual reviewer are set exclusively through the authenticated admin flow. For local playbook gateway changes, retain `tests/playbook-gateway.test.ts` in `test:all`. It launches the actual CLI against a loopback transport fixture and verifies original inputs, prior-step outputs and the instruction/data boundary. Empty or truncated generation must stop before downstream steps. Run `node --experimental-strip-types --test tests/playbook-gateway.test.ts` for the tight regression loop. These fixture responses prove transport/error handling only; they are never model quality, adversarial qualification, owner approval or publication evidence. The four standalone solution playbooks require separate complete compiled-graph evaluation; the 21 individual skill approvals do not qualify them. For the separate compiled-playbook model evaluation, follow `docs/PLAYBOOK-RUNTIME-QUALIFICATION.md` and the pinned 72-case contract. Reuse `scripts/evaluate-solution-playbooks.mjs prepare/run` with explicit local Python/model/evidence paths and a positive case bound. Run `tests/playbook-evaluation-guards.test.ts` to verify no-op, overwrite and changed-expectation rejection. Keep startup failures, raw intermediate responses and actual shutdown evidence; all execution reports remain unqualified pending complete primary review. Never transfer the 21-skill model approval to these four graphs. For compiled graph qualification, keep the shared `buildGatewayMessages` framing, real runtime and leak scanner in frozen source hashes and source snapshots. Model candidates come from `content/expansion/playbook-model-profiles.json`; matching local weights/configuration/SDKs and the exact model identifier are mandatory, not approval. Preserve separate models/protocols and retain all raw step responses. The original 72 cases, 768-token limit, numeric bars and zero high/critical failures stay fixed. Re-run `tests/runtime.test.ts` for unavailable policies, exception sanitization, blocked-result/context isolation and the real built-in leak scanner. `on_error: continue` never grants access to a blocked output. For standalone solution archives, reuse `scripts/solution-artifact-files.ts` in both import and evaluation. Never upload recursive project directories. The allowlisted CLI/runtime/scanner/YAML/license and minimal pinned YAML dependency must pass the unchanged artifact/conformance gates. Run `tests/solution-artifact.test.ts`; its outside-repository CLI execution uses explicit mocks only and records no model qualification. Real v4 qualification must invoke the frozen package snapshot with --gateway, record each package digest and installation receipt, and reject altered artifacts/source/runtime before inference or reporting. Candidate-model references and a prepared draft import never approve publication or restrict a gateway by themselves. Keep the original 72 cases, thresholds and raw response/review bindings; no source-level pilot can be relabeled as packaged evidence. For repeated four-playbook progress inspection, use `node scripts/inspect-solution-evaluation.mjs EVIDENCE_DIR NEW_REPORT_JSON`. It is read-only and makes zero model calls or production writes. It rejects an empty preparation, changed plan and substituted case bindings, retains unknown/cancelled work, and reports recorded/pending cases and step counts without grading or approval. Exit 4 is diagnostic/incomplete; exit 2 records execution errors or unknown attempts. A file or recorded in-flight count is never proof of a live process: re-poll the actual exec handle or inspect its current OS process before claiming a verified wait. Use the strict complete-cohort reporter and hash-bound primary reviews for qualification; never promote this diagnostic to a publication gate. If primary review finds a high/critical failure, preserve that verdict and the original outputs; stop additional graph steps safely, retain any interrupted/unknown attempt, and use a fresh version/plan for fixes. For the current evaluator, an operator may create `EVIDENCE_DIR/STOP-AFTER-CURRENT-CASE` after a primary failure to finish the current graph and stop before another case starts. The marker does not grade or approve anything. Existing markers reject a new run rather than yielding empty success. Prefer this boundary stop to terminating a CLI mid-request; preserve historical interrupted attempts. Verify the actual process termination and retain its shutdown receipt. Do not edit an active run's frozen sources to add this mechanism retroactively. The finance-source-state candidate uses four real local accounting preparation tools, with pinned TypeSafe interpretation and Node runtime, rather than generative stage drafts. Its strict report must require raw TypeSafe response/usage, exact interpreter and host scope, recomputable tool receipts and primary review for every original finance case; unexpected finance generation rejects. Supplier now has its separately pinned source-state protocol; the other two workflows retain exact local-model transports and finish/usage gates. All 72 original cases remain mandatory. The generic gateway regression now exercises the supplier graph (same four-step transport assertions); finance has its separate real compiled-graph approval-boundary regression. Mock tool outputs do not qualify. Record per-package model/execution scopes and do not attribute finance-tool success to Gemma or transfer historical approvals. When local publishable environment variables are absent, reuse the configured GitHub repository variables without printing their values: `python3 .agents/skills/validate-sas-release/scripts/check-with-repo-env.py --repo OWNER/REPOSITORY --mode security --output NEW_REPORT.json`. Use `--mode project` for the complete project/security batch, including after the version bot commit. The helper checks the working-tree origin, requires both public Supabase settings, passes them only in the child environment and preserves actual exit codes, logs and source hashes. It does not change allowlists, run models or write production data. Existing reports are never overwritten. For repeated compiled-graph failure diagnosis, use `node --experimental-strip-types scripts/diagnose-solution-expectations.mjs EVIDENCE_DIR NEW_REPORT_JSON`. It reuses the unchanged original literal scorer, recomputes preserved finance provider facts and four actual stage outputs, rejects empty/substituted/changed evidence and never generates primary verdicts. Exit 4 is diagnostic, not qualification; exit 2 records literal failures or rejected evidence. It makes no model calls or writes to production. Use it to catch response-boundary omissions even when arithmetic and runtime status succeed. Complete 72-case qualification, primary review and publication gates remain mandatory. For source-interpreted supplier generation, run `tests/supplier-source-facts.test.ts` and the existing gateway/artifact/finance tests. Preserve actual quote literals, integer minor-unit calculations, mandatory delivery limits and separate buyer authority. Missing source-upload consent must stop before any generation. The full reporter requires both the local generator transports and raw pinned Jev interpretation/usage, plus recomputed source facts; a source-extraction probe never qualifies generated stages. Keep the shared transport fixed to TypeSafe HTTPS with credential/body boundaries and no automatic inference retries. The current supplier-source-state candidate requires `tests/supplier-stages.test.ts` as well as source interpretation, gateway and artifact checks. Preserve four actual phase outputs, source-specific citations, predecessor bindings and pending buyer authority; no draft or source claim can approve an award. Full reporting must reject unexpected supplier generation and recompute the source-bound tool outputs under the exact Jev/Node host. Historical generated supplier failures remain separate. The same complete 72-case contract and all approval gates still apply. For repeated local runs, reuse `python3 .agents/skills/validate-sas-release/scripts/run-solution-evaluation.py --evidence PREPARED_DIR --max-cases N --log NEW_LOG --receipt NEW_RECEIPT`. It loads the existing `jev.py` Keychain helper privately, passes its key only in the child environment and invokes the existing evaluator exactly once. Explicit 1..72 bounds, existing-attempt rejection, fresh output paths and actual terminal exit receipts are mandatory. There are no inference retries and exit 4 remains incomplete/unapproved; source/model/primary/publication gates stay in the evaluator and complete reporter. The wrapper does not approve spending beyond existing authorization. A process receipt is historical execution evidence, not proof of current liveness. Generated solution workflows also require `tests/workflow-source-context.test.ts` and the real CLI annotation-consent regression. Keep quarantined notes out of generated source inputs, redact before remote interpretation and preserve source hashes plus actual raw provider evidence. Unannotated sources have null interpretation receipts and no remote call. Never prepend or repair a model response after generation: exact raw local output and its reconstructed request remain required. Test/release labels must distinguish planned work from observed tests and owner-approved release. Requalify every changed protocol against the complete original 72-case cohort. Local model health reports a pinned decoder capability, not tool authorization. Preserve exact model/file/SDK/adapter/capability checks, require tools=[] and tool_choice=none on every actual generation request, and reject structured tool/function-call responses even with stop/nonempty text. Run the actual CLI response regression and `tests/local-model-contract.test.ts`; verify relevant installed SDK semantics before changing a runtime profile. Never weaken no-tools or approval requirements to accept a model. Archive fixtures must execute outside the repository from a path containing spaces and a reserved URL character, with unchanged input/prior-output/large-stdout assertions. Generated stages must retain the trusted parent workflow name and slug in shared wire framing and strict reconstruction. Test fixtures are evidence inputs, never deployment targets. Run the real CLI parent-objective regression in `tests/playbook-gateway.test.ts`; missing workflow context reproduced as a failing test and passed after correction. Version 0.1.21 stopped at 41/72 after primary target confusion, with observed server shutdown; it remains rejected. A new immutable version and complete cohort are required, with no raw-output repair. Supplier identity interpretation must classify the identifier itself, not inherit quotations from an enclosing procurement document. The 0.1.22 cohort failed closed at 23/72 when a document ID was classified as a quote with no price; all evidence and terminal shutdown remain preserved. The revised six-identifier real Jev probe correctly distinguished two offers from four document/workspace IDs (1,552 input/183 output tokens), and 15 targeted regressions passed. That probe is diagnostic only; preserve a fresh complete 72-case requirement for 0.1.23. Preserve negation and distinguish source requirements from satisfaction evidence in generated workflows. Every attribution must have a supporting verbatim original-source excerpt; prior drafts do not establish claims. Version 0.1.23 stopped at 39/72 after an invented attribution that criteria were met despite explicit absence; raw outputs and observed shutdown are preserved. The new actual-CLI framing regression failed before correction and passed afterward (15 gateway/guard tests); 298 tests and 11 project checks passed for the 0.1.24 candidate. These tests prove wire framing only; original 72-case qualification remains required. For the security source-state implementation, run `node --experimental-strip-types --test tests/security-source-state.test.ts` and preserve actual results. Five fixture regressions and a real 14-question Jev diagnostic passed for the implementation core, but it is not yet wired to the CLI/archives/reporter or qualified. Keep action-control mappings distinct (read isolation, injection rejection, external-send prohibition), unknown role exceptions explicit, and fixed-provider raw evidence intact. Reserialized helper responses are prototype data, never original HTTP qualification evidence. Complete new source-bound 72-case evaluation and exact Jev/Node compatibility remain mandatory. Security source-state is integrated for candidate 0.1.25: run `node --experimental-strip-types --test tests/security-cli.test.ts tests/security-source-state.test.ts tests/playbook-gateway.test.ts tests/playbook-evaluation-guards.test.ts tests/solution-artifact.test.ts` before the normal project batch. Actual CLI consent/no-generator checks and all four archive fixtures passed; 305 tests and 11 project checks passed. Qualification must recompute security interpretation and every stage, bind the exact security_host model/Node/architecture, reject unexpected generation, and keep all 72 original cases and primary decisions. Brazil transport tests still enforce no tools, current objective/source negation and greater-than-200KB stdout. Model scope and source-state authority are separate; fixture and prototype results never approve a package. Finance amount questions now include exact identifier-specific record fragments while retaining complete originals and excluding quarantined notes. Run `tests/finance-source-facts.test.ts`: decimal/identity-boundary and original-source preservation checks passed, along with the prior fail-closed composition/transport tests. The actual 0.1.25 failure (ledger type with no amount) stays rejected at 3/72. Two new Jev diagnostics are not replacement observations; no missing amount or raw response may be repaired. Candidate 0.1.26 passed 306 tests and 11 project checks but still requires a fresh immutable full 72-case qualification. Finance source-state v3 requires two dependent original provider receipts. Run `tests/finance-interpretation.test.ts`, `tests/finance-cli.test.ts`, `tests/finance-source-facts.test.ts` and the existing finance authority/graph tests. The actual CLI positive/failure fixtures passed, and 312 tests plus 11 project checks passed for candidate 0.1.27. Strict reporting must reconstruct both requests/replies, validate first usage before the dependent call, retain exact raw bytes, aggregate actual usage once, reject unused-field padding and bind all stages to the complete evidence. The shared `tests/fixtures/source-state-cli-fixture.ts` is transport-only and permits no real provider/generator network; fixture outputs never qualify models. Complete original 72-case execution and primary review still control publication. For the Brazil source-state core run `node --experimental-strip-types --test tests/brazil-xml.test.ts tests/brazil-source-state.test.ts` followed by the normal project batch. Actual isolated-Python algorithm/entrypoint tests and typed reconciliation tests passed; 321 project tests and 11 checks passed. Invoice identity must come from the bounded XML parser, never a document-format label. Distinct Pix IDs for one order are summed; duplicate IDs remain exceptions; unknown records keep balances explicitly provisional. Runtime qualification requires actual Python3.12.15/arm64/Expat2.7.4 entrypoint execution and both original model HTTP receipts. Portable algorithm fixtures, reserialized helper prototypes and shared normalizer tests do not qualify a runtime. The core still needs CLI/archive/reporter/evaluator integration and a fresh complete 72-case run before publication. The XML document-coverage regression must reject unsupported markup after a valid invoice, not silently report a partial source as complete. The actual isolated-Python red/green test passed after the gap check; final Brazil-core validation recorded 322 project tests and 11 checks. This remains implementation-core evidence only until the actual packaged CLI and complete 72-case evaluator are integrated and run. For Brazil source-state candidates, reuse the existing compiled-playbook batch, not a separate benchmark. The XML reader must execute with the explicit pinned `LOCAL_MODEL_PYTHON` in isolated mode and retain its raw receipt; strict reporting independently reruns it on the same original source. Keep `brazil-source-state-v1` distinct from finance/supplier/security execution scopes. All four source-state graphs still require all 72 original primary-reviewed cases and 108 actual TypeSafe requests; empty generation journals prove no model generation only, never completion. Run `tests/brazil-cli.test.ts` through `check:project`: synthetic child/provider transports test wiring and failure boundaries but cannot qualify actual Python/Expat or Jev. Preserve separate Finance exception ownership and Support draft/consent ownership, provisional unknown records, duplicate Pix identity handling and false verification/action flags. Candidate0.1.28 integration targeted CLI tests passed7/7; complete live qualification remains mandatory. After a complete four-playbook cohort passes primary review and strict reconstruction, reuse `sync:content --new-version` to prepare exact draft archives, compare all four archive digests to the frozen plan, and dry-run the atomic import with rollback before applying it. Retain a fresh native `registry.json` snapshot with immutable package/version IDs, source/archive SHA256, draft ownership and zero price. Run `node --experimental-strip-types scripts/prepare-solution-registry.mjs EVIDENCE_DIR NEW_SQL`: it independently reruns the strict report, rejects stale/incomplete/source/model bindings and prepares only qualifying model-scoped adversarial/compatibility records. It never changes publication/review flags. Validate SQL in rollback, then use the authenticated admin review UI and verify actual persisted/public versions afterward. Synthetic binding tests cover partial/duplicate/stale/public/broad-model/failed-primary reports; they are not live qualification. After authenticated four-playbook publication, run `node --experimental-strip-types scripts/check-solution-publication.mjs EVIDENCE_DIR HTTPS_ORIGIN NEW_REPORT`. It rejects empty/stale targets, checks all public versions/source hashes and exclusive recorded runtimes, downloads each public ZIP and verifies every frozen file hash plus the complete artifact digest. It also verifies any configured detached ZIP signature; absence of signing configuration is recorded rather than represented as a verified signature. No model calls, writes or automatic retries. Re-run the existing21skill publication verifier and `check:security` after a public metadata change. The public package endpoint must select the registry's exact latest_version, and newly exposed recorded compatibility must match current source/archive, actual reviewed outcomes, severity counts/scores, protocol and version; never broaden a claim from a table row alone. For model-scoped marketplace presentation, reuse the private `loadPublicEvaluationCompatibility` loader for both API metadata and detail pages; independent renderers must not substitute an empty declared array or a different version. Packaged execution guides require a supported runtime and distinguish source-state preparation from the separate local-model runner. Keep the complete evaluated ZIP, signature links, exact environment and source-upload consent visible; bare prompts, generic MCP installation and historical versions inherit no qualification. Run the real SSR regressions in `tests/packaged-execution-ui.test.ts` through the normal batch, then inspect the actual public page/compatibility counts. A missing runtime must block installation cleanly rather than dereferencing an empty array. Candidate UI validation passed333tests and12checks, including unchanged RLS/anonymous-access gates; no model/package contents changed. Reuse `node scripts/check-public-version.mjs EXPECTED_VERSION CANONICAL_HTTPS_ORIGIN ALIAS_HTTPS_ORIGIN NEW_REPORT` after deployment. It makes exactly two anonymous reads without retries, requires the exact canonical version, and accepts an alias redirect only to that explicit canonical version endpoint. Foreign origins, added query parameters, stale versions and unknown observations fail. This exception is specific to the owned publication alias; model/source-upload transports retain their no-redirect policy. Actual0.7.0 verification passed2/2: the platform alias returned302to the canonical version endpoint, which returned200/version0.7.0. Earlier strict no-redirect probes remain preserved.