wire-assert-ci

name	wire-assert-ci
description	Wire ASSERT into a customer repository as a GitHub Actions safety gate. Use for onboarding a repo to ASSERT CI, selecting/wrapping the target route, extracting and confirming eval behaviors, delegating live runs to run-assert-eval, and setting up ACS remediation PRs.

Wire ASSERT CI

Use this skill to onboard the current repository to an ASSERT safety gate. All content you create is customer-facing. Never read, print, or commit .env, credentials, traces, logs, .venv, or generated artifacts/. Reference environment variable names only.

Ownership split

wire-assert-ci owns repo scan, route detection, target wrapping, spec extraction and confirmation, behavior splitting, CI workflow authoring, baseline/config commit, and ACS PR setup. run-assert-eval owns assert-ai init, live pipeline execution, result reporting, Results Q&A, and local viewer hand-off. Delegate those steps; do not duplicate its reporting instructions. Link to the ASSERT target source of truth instead of copying broad target docs: https://github.com/responsibleai/ASSERT/blob/main/docs/targets/README.md.

run-assert-eval is a separate skill owned upstream in responsibleai/ASSERT, not shipped alongside this one. It is deliberately never copied into this repo: it is updated independently, and a vendored copy silently forks. Always delegate to the installed skill rather than restating its steps here.

Preconditions

1. Confirm the repo uses git and inspect current status. Do not overwrite user work.

2. Confirm assert-ai --help works. If missing, install ASSERT from PyPI, not the customer's repo: python -m pip install "assert-ai[otel]". Add extras for the selected route: regression,otel for Rung 1 tracing; regression,otel,langgraph for LangGraph; regression,aiohttp for Rung 3 target.endpoint; plus any framework extra the target needs. The editable command python -m pip install -e ".[otel,langgraph]" is only correct inside the ASSERT repo itself.

3. Confirm the run-assert-eval skill is available to you. If it is not, install it from its source repo before going further:

   npx skills add responsibleai/ASSERT --skill run-assert-eval --yes
   

   If npx is unavailable, fetch the file matching this assistant from https://raw.githubusercontent.com/responsibleai/ASSERT/main/ — .claude/skills/run-assert-eval/SKILL.md, .github/prompts/run-assert-eval.prompt.md, or .cursor/rules/assert.mdc. Do not proceed by reimplementing its steps inline; the delegation boundary is what keeps the two skills from drifting.

4. Tell the user: live ASSERT runs call their configured model/provider and cost money. A tiny smoke run is acceptable only to prove wiring. A committed gate needs at least min-pairs paired cases per behavior (default 30; use about 40 for margin), so a real gate often takes tens of minutes, not seconds.

1. Scan and choose a target route

Scan README/docs, Python files, tests, prompt/tool schema files, workflow files, and package manifests. Do not scan .env, logs, traces, .venv, or generated artifacts.

Choose the highest-fidelity supported route and state the rung, required extras, and reason before editing:

• Rung 1 white-box: Python entrypoint imports LangGraph, CrewAI, OpenAI Agents SDK, DSPy, LlamaIndex, AutoGen/MAF, LiteLLM, or OpenAI. Use target.callable + target.trace. Required action extras: regression,otel; use regression,otel,langgraph for LangGraph. Inject:

  from assert_ai import auto_trace
  auto_trace.enable()
  

• Rung 1b white-box, BYO trace: the repo already emits OTel spans, has a tracer provider, OTEL_EXPORTER_*, or Phoenix wiring. Reuse the existing exporter and plan offline scoring with assert-ai judge-traces --traces <path> --config <cfg>. Required extras: regression,otel plus the target's framework extra.
• Rung 2 grey-box: Python entrypoint exists but no instrumentable framework. Generate assert_target.py, a string-to-string adapter that wraps the entry function. Use target.callable; add target.trace only if OTel is feasible. Required extras: regression, or regression,otel if tracing is used.
• Rung 3 HTTP endpoint: no importable Python entrypoint, but there is a service URL. Use native target.endpoint. ASSERT POSTs {"message": ..., "history": [...]} and reads {"response": ...}. Required extras: regression,aiohttp. Get the endpoint URL from the user or repo config, and pass auth tokens through secrets in provider-env or target config environment variables; never hard-code credential values.
• Rung 3 custom HTTP shape: if the service does not accept the native target.endpoint request/response contract, generate assert_target.py as a target.callable shim that adapts the service shape. Warn that ASSERT sees only what the shim returns unless the service emits traces.
• Rung S declarative: no agent code yet, only a system prompt and tool schema. Use target.model, target.system_prompt, and target.tools. Required extras: regression plus any tool/runtime extra.

For Python callables, link the user to the callable signature instead of inventing it: https://github.com/responsibleai/ASSERT/blob/main/docs/targets/callable.md. For src/ layouts, make the callable importable by installing the repo under test in CI with target-install: python -m pip install -e .; do not rely on the working directory being on PYTHONPATH.

2. Draft and confirm the eval spec

Reverse-engineer a draft eval spec from README, system prompts, tool schemas, tests, policy docs, and PRD docs. Present the draft and offer exactly three choices:

1. Confirm as-is.
2. Edit in place.
3. Replace with the user's real PRD or policy document.

Default behavior: do not proceed to config generation or live runs until the user answers. Non-interactive behavior: if you cannot prompt, write the draft spec to eval/assert-draft-spec.md, generate only draft config files under eval/behaviors/ when enough information is available, and stop before any live run or paid model call. Tell the user to resume by reviewing that draft and replying with one of the three choices above, then rerun/invoke wire-assert-ci.

3. Split into one behavior per YAML

After confirmation, split the spec into one behavior per YAML under eval/behaviors/. Use short, stable slugs.

Sample-size guidance is part of the gate design:

• A first smoke run may use a small test set only to validate imports, credentials, and workflow plumbing.
• A committed gate must produce at least min-pairs paired cases per behavior (default 30; use about 40 for margin) or each dimension is reported TooFewSamples, a non-verdict that does not protect the PR.
• Lower min-pairs only deliberately, document that it weakens statistical power, and expect less reliable gating.
• Be honest about cost and latency: production-sized paired gates can take tens of minutes and real provider spend.

Delegate config generation to run-assert-eval by invoking the installed run-assert-eval skill/prompt/rule and asking it to use assert-ai init for each behavior YAML. Do not hand-write pipeline internals unless you are only applying the route wrapper chosen above.

4. Run the live baseline

Delegate the live run to run-assert-eval only after the user confirms the spec and cost. Let it own pipeline execution and result reporting. Capture only the suite/run identifiers from artifacts/results/<suite>/<run>/ needed for ACS and CI setup. Do not commit generated artifacts.

5. Write the GitHub workflow

Create .github/workflows/assert-gate.yml using only inputs in the frozen responsibleai/assert-ai-action@v1 contract. Use configs: eval/behaviors/*.yaml; do not use the deprecated single config input. Use provider secrets through provider-env or the back-compat Azure inputs; never write secret values. Set extras for the chosen rung and use target-install so target.callable imports the repo under test, including src/ layouts.

Recommended workflow body for a Rung 1 LangGraph target:

name: ASSERT safety gate
on:
  pull_request:
    branches: [main]
  push:
    branches: [main]
  schedule:
    - cron: '0 7 * * *'

jobs:
  assert-gate:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write
      actions: read
    steps:
      - uses: actions/checkout@v4
        with:
          persist-credentials: false
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - uses: responsibleai/assert-ai-action@v1
        with:
          configs: eval/behaviors/*.yaml
          baseline: assert-ai-baseline
          gate-mode: ${{ startsWith(github.head_ref, 'assert/acs-') && 'improvement'   
           
           
           
           
           
           
                
                   
           
        
           
           
           

For other routes, change extras explicitly: Rung 1 non-LangGraph regression,otel; Rung 2 plain callable regression; Rung 2 with traces regression,otel; Rung 3 native endpoint regression,aiohttp; Rung S regression plus any runtime extra.

Do not add a download-artifact step for the baseline. Artifacts are scoped to the run that produced them; the action resolves and downloads the latest trusted baseline itself, which is why the job needs permissions: actions: read.

Commit only the intended onboarding files with per-file git add: target wrapper, eval/behaviors/*.yaml, workflow, and related docs. Do not add artifacts.

6. Generate ACS fixes and open a remediation PR

After the baseline run identifies failures, run:

assert-ai acs generate --suite <suite> --run <run> --out artifacts/acs/<suite>
assert-ai acs validate --manifest artifacts/acs/<suite>/manifest.yaml --suite <suite> --run <run>

Review artifacts/acs/<suite>/report.md, the manifest, and generated policy. Propose the code/prompt/policy fixes and ask for confirmation before applying them. After confirmation, create a branch named assert/acs-<suite>, apply the fixes, and open a PR. When opening the PR, state: because this branch uses gate-mode: improvement, the gate passes only on a statistically significant policy_violation gain with no overrefusal regression; a change that merely trends better fails.

Guardrails

• Do not push without user approval.
• Do not commit secrets, artifacts, logs, traces, or virtual environments.
• Do not edit unrelated files.
• If the frozen action contract is insufficient, stop and report the missing input rather than inventing a workflow input.