agilab-deep-audit

name	agilab-deep-audit
description	Produce deep AGILAB audit and code-review artifacts with evidence-backed findings, mandatory architecture-foundation readiness, blast-radius tracing, security/test posture, and prioritized recommendations. Use when the user says "review AGILAB", "audit AGILAB", "code review AGILAB", "deep review", "architecture review", "security review", asks for a review document, or asks for comparison-quality critique rather than a quick fix.
license	BSD-3-Clause (see repo LICENSE)
metadata	{"updated":"2026-06-24T00:00:00.000Z"}

AGILAB Deep Audit Skill

Use this skill when the user asks for an AGILAB audit or a review artifact. Do not treat an audit request as a maintenance/fix request unless the user explicitly asks to patch findings.

Operating mode

• Start in review mode, not implementation mode.
• Inspect current repository state before making claims; do not rely on memory for branch, version, workflow, or public-posture facts.
• Load references/ARCHITECTURE_FOUNDATIONS.md before a deep audit unless the user explicitly requests a narrow file-only review. If the session cannot explain the foundations in that reference from current repository evidence, pause the final audit and perform an architecture read pass first.
• State scope and limits explicitly. If the repo is too large for exhaustive reading, say which central modules were read and which periphery was sampled.
• Build a thesis, not a checklist. Identify the dominant root cause or design pressure when findings cluster.
• Findings come first for code reviews. For broader audits, put verdict and executive summary first, then detailed evidence.
• Do not overclaim. If a module or workflow was not inspected, label the point as an inference or residual risk.

Architecture readiness gate

Before producing a final deep audit, confirm the session has current evidence for AGILAB's founding principles:

• Trusted-operator reproducibility workbench, not a standalone production MLOps control plane.
• Three-plane model: control plane, payload plane, evidence plane.
• Lean base package plus optional extras and split packages for heavier capabilities.
• Built-in apps can own project-specific dependencies; generic apps-pages and the agi-pages umbrella must remain app-agnostic and avoid project-specific runtime dependencies.
• Linux, macOS, and Windows support is part of the public posture unless a feature states a narrower boundary.
• Generated code, notebooks, external apps, workers, page bundles, and cluster execution are executable-code trust boundaries.
• Docs, release proof, package split, and public claims must agree before a feature is described as shipped.

If any point is unclear, do not guess. Read the relevant package manifests, provider registries, docs, release proof, tests, and workflow files first, then state what remains unverified.

Default AGILAB audit scope

For a deep AGILAB audit, cover these surfaces unless the user narrows scope:

• Package topology: root pyproject.toml, core package manifests, extras split, and stable handoff claims.
• Core runtime: agi-env, agi-node, agi-cluster, agi-core, especially singleton state, command execution, worker dispatch, local/remote deployment, path resolution, and service/evidence helpers.
• Execution engine: dispatcher/distributor scheduling, chunking, worker capacity, generated code paths, shell boundaries, and runtime dependency mutation.
• UI and public exposure: Streamlit entrypoints, public-bind guard, page state, session-state boundaries, and user-visible workflow paths.
• Security posture: credential redaction, subprocess/shell usage, pickle or executable payload boundaries, external apps policy, public UI controls, supply-chain/provenance, and MCP write/exec exposure. Also check SSH host-key verification and auth mode in cluster transport, remote command construction (exec_ssh argv vs interpolated strings), connector SSRF (scheme allowlist, metadata-range blocking, no credential forwarding), MCP read-tool path containment, and html.escape on values rendered via unsafe_allow_html.
• Cross-platform and tests: Windows/macOS/Linux claims, hermetic HOME behavior, singleton resets, path separators, process/signal assumptions, and CI workflow coverage.
• Benchmark and performance evidence helpers: verify that scripts and docs declare their exact topology and platform boundary. A lab-specific helper such as a macOS-over-SSH benchmark must not be treated as portable cluster, release, or production evidence unless the code fails closed outside that boundary and tests/docs name the limitation.
• Packaging and docs: wheel surface, generated artifacts, docs/public mirror, release proof, and changelog/public claim alignment.
• Periphery sampling: representative built-in apps, page bundles, and examples only after the central architecture is understood.

Production and evidence-readiness audit mode

Use this mode for high-value AGILAB production and evidence-readiness audits. It is a reusable review contract, not a copied chat prompt or a generic bug hunt.

Audit scope: full AGILAB repo, docs, release proof, package split, built-in
apps, notebooks, MCP and agent surfaces, Streamlit pages, tests, generated
assets, Hugging Face/PyPI/GitHub publication state, current git/GitHub state,
and commit provenance when authorship or attribution matters.

Reviewer stance: senior production engineering, open-source product strategy,
security review, and evidence/reproducibility readiness.

Review goal: make AGILAB production-grade as a trusted-operator workbench for
turning AI/ML experiments, notebooks, and agent runs into replayable,
attestable evidence. Keep claims honest: do not position AGILAB as a fully
managed production MLOps control plane or regulatory compliance product unless
current public evidence proves it.

Risk areas:
- correctness bugs, performance hot paths, and validation cost
- broken or misleading user workflows
- public/private boundary leaks, unsafe executable-content handling, credential
  exposure, and dependency/provenance gaps
- wrapper, MCP, agent-skill, generated-discovery, and command-routing drift
- evidence, proof, replay, release-proof, and artifact-hash misalignment
- package split, optional extras, built-in app, and app-contract drift
- docs/code/changelog/README/PyPI/Hugging Face/GitHub state misalignment
- test gaps around risky behavior, polluted environments, and trust boundaries
- release/build/reproducibility/SBOM/provenance problems
- ambiguous commit provenance, including mismatches across author, committer,
  signature identity, GitHub actor, PR metadata, and timestamps
- newcomer confusion and open-source product-positioning gaps
- places where AGILAB's trusted-operator, replayable-evidence story is not
  clear enough

Follow-up workflow:
1. List findings by impact.
2. Fix the top actionable issues directly when the user asks for fixes.
3. Add or strengthen regression coverage or durable guards.
4. Regenerate required docs/assets/discovery artifacts.
5. Run the strongest practical local validation suite for the touched scope.
6. Commit, push, and use the repo PR/merge policy for the reviewed batch.
7. End with a concise report: fixed, validation, remaining risks, next
   highest-value move.

Execution rules:

• Run the prompt as review first; do not patch unless the user asks to fix findings.
• If using a subagent, pass the prompt and repository target without leaking expected findings or intended fixes.
• When fixing findings, prioritize deterministic issues with a narrow regression, guard, or generated-artifact check.
• Keep product language aligned with AGILAB's public evidence posture: replayable and attestable experiment evidence, not overclaimed enterprise production automation.

Multi-axis parallel review mode

When the user asks for a Claude-style multi-axis review, a 7 axis pass, or asks to inspect pitfalls/speed/correctness/known bugs/source tree/naming/ deprecations/security/docs alignment/repo status, split the read plan across the current review lanes below and merge their evidence before writing findings. Treat 7 axis as historical shorthand for this AGILAB multi-axis mode; AGILAB reviews still include security, docs-alignment, and repo-status lanes unless the user explicitly narrows the scope. Parallelism here means parallel evidence collection and comparison, not parallel edits or speculative fixes.

1. Pitfalls: hidden state, trust-boundary mistakes, environment pollution, fragile ordering, missing fail-closed behavior, and traps already encoded in AGENTS.md or AGENT_LEARNINGS.md.
2. Speedup: startup time, validation cost, runtime hot paths, log/token volume, avoidable dependency syncs, and opportunities to parallelize reads or batch work without changing semantics.
3. Correctness: invariants, data-path resolution, scheduler/worker behavior, cross-platform determinism, error handling, and whether tests prove the intended contract instead of only the happy path.
4. Known bugs: match the current diff or audited surface against nearby TODO/FIXME comments, regression tests, issue-shaped guardrails, historical bug-class rules, and release/changelog notes; do not report a remembered bug unless current code still shows the mechanism.
5. Source-tree improvement: duplicated helpers, misplaced generated files, package-boundary drift, docs/source mirror confusion, stale artifacts, and opportunities to move code to the existing owner instead of adding a parallel mechanism.
6. Naming: user-visible labels, config keys, public APIs, app/page names, evidence field names, and test fixtures that could drift semantically or conflict with cross-page action semantics.
7. Deprecation: deprecated Streamlit/Python/package APIs, retired AGILAB helpers, legacy build wrappers, obsolete install paths, and old docs claims that should be migrated, shimmed deliberately, or removed.
8. Security: trust-boundary violations, untrusted content handling, shell/subprocess execution, credential or token exposure, public UI binding, MCP/LLM tool exposure, dependency supply chain, and provenance or artifact verification gaps.
9. Docs alignment: mismatches between code behavior, user-visible UI, README/public docs, canonical docs source, mirrored docs, screenshots, SECURITY.md, CHANGELOG.md, release proof, and public evidence claims.
10. Repo status: current branch, upstream divergence, staged and unstaged changes, untracked non-ignored files, ignored generated byproducts that could pollute guards, recent commits, author/committer/signature/GitHub actor provenance when relevant, open PR/check state, and whether the checkout is safe for review, patching, push, release, or merge.

For each lane, keep evidence concrete: cite files, line ranges, commands, tests, or public artifacts. After the lane read, deduplicate overlapping observations into root-cause findings and explicitly mark any lane with no actionable finding as clear rather than padding the report.

Evidence collection pattern

Plan the read pass before opening files. Prefer targeted reads over broad scans:

git status --short --branch --untracked-files=no
git log --oneline -5
git log --show-signature --format=fuller -5
sed -n '1,220p' pyproject.toml
find src/agilab/core -maxdepth 3 -name pyproject.toml -print
rg -n "class AgiEnv|__new__|__init__|shell=True|pickle\\.load|eval\\(|exec\\(|subprocess|create_subprocess|AGILAB_PUBLIC_BIND|MCP|FastMCP|security-check|Path\\.home|except Exception|bare except|uv add|command -v sshfs|known_hosts|StrictHostKeyChecking|asyncssh|sshpass|exec_ssh|urlopen|unsafe_allow_html|is_relative_to" src/agilab tools test

Then read the specific files implicated by the hits. For full audits, useful anchors are usually:

• src/agilab/core/agi-env/src/agi_env/agi_env.py
• src/agilab/core/agi-env/src/agi_env/execution_support.py
• src/agilab/core/agi-node/src/agi_node/agi_dispatcher/base_worker.py
• src/agilab/core/agi-node/src/agi_node/agi_dispatcher/work_dispatcher.py
• src/agilab/core/agi-node/src/agi_node/agi_dispatcher/build.py
• src/agilab/core/agi-cluster/src/agi_cluster/agi_distributor/api/agi_distributor.py
• src/agilab/core/agi-cluster/src/agi_cluster/agi_distributor/deployment/deployment_local_support.py
• src/agilab/core/agi-cluster/src/agi_cluster/agi_distributor/deployment/deployment_remote_support.py
• src/agilab/core/agi-cluster/src/agi_cluster/agi_distributor/runtime/transport_support.py (SSH/SCP host-key, sshpass, exec_ssh remote commands)
• src/agilab/lab_run.py
• src/agilab/security/ui_public_bind_guard.py
• src/agilab_mcp/server.py when MCP exists (read-only evidence tools; check path containment in src/agilab_mcp/manifest_tools.py)
• test/conftest.py, root tests, and src/agilab/core/test
• README.md, SECURITY.md, CHANGELOG.md, and docs/source/release-proof.rst

When reading line-sensitive code, use nl -ba <file> | sed -n 'start,endp' or an equivalent command so the final review can cite stable line references.

Review quality bar

Each important finding should include:

• Severity: CRITICAL, HIGH, MED-HIGH, MED, LOW
• Evidence: file and line references, concrete call-sites, commands, or test failures
• Mechanism: what the code actually does
• Impact: what can break, leak, become nondeterministic, or mislead users
• Blast radius: which pages/apps/workflows/tests are affected
• Recommendation: the smallest good fix plus any stronger architectural fix
• Regression plan: targeted tests or workflow profiles that would prove the fix

Prefer one strong root-cause section over many shallow bullets when the same design issue appears in several places.

Output template

For a review document, use templates/CODE_REVIEW_TEMPLATE.md as the canonical file structure. If the template file is unavailable, write this structure:

# AGILAB — Detailed Code Review

> Reviewer: Codex · Date: YYYY-MM-DD · Branch: `<branch>`
>
> **Scope.** <what was read fully, what was sampled, and what was out of scope>

## Executive summary

<verdict, dominant thesis, strongest positives, and top risks>

## 1. Architecture & module topology

<package graph, stable handoff assessment, module decomposition, drift/debt>

## 2. High-severity finding title · **HIGH**

<mechanism, evidence, call-sites, blast radius, recommendation>

## 3. Execution engine

<scheduler/dispatcher/build/runtime mutation findings>

## 4. Security posture

<public bind, shell, secrets, pickle, MCP, external apps, supply chain>

## 5. Testing & cross-platform

<test breadth, hermeticity, Windows/macOS/Linux claims, CI coverage>

## 6. Packaging & repo hygiene

<wheel surface, generated artifacts, docs/release/public claim alignment>

## 7. Prioritized recommendations

| # | Severity | Issue | Action |
|---|---|---|---|
| 1 | **HIGH** | <issue> | <action> |

## Bottom line

<short, honest final judgement>

For a shorter audit response, keep the same logic but compress it: verdict, scope, top strengths, top findings, prioritized actions, bottom line.

Quality gate

When the audit is written to a Markdown file, run the deterministic quality evaluator before handoff unless the user explicitly asks not to validate:

uv --preview-features extra-build-dependencies run python tools/audit_quality_evaluator.py <audit.md> --min-score 80

Use ./dev audit-preflight before writing a long audit when the architecture context is not fresh. Use ./dev audit-quality <audit.md> as the short local gate after writing the audit.

Use --json --output <report.json> when the score should become evidence. A score below 80 means the artifact is not yet comparison-quality; improve the missing rubric areas before presenting it as a final audit.

Validation and follow-up

• Do not patch during the audit unless the user asks.
• If the user later says fix it, convert the prioritized findings into a narrow implementation plan and use agilab-testing to select regressions.
• If the audit is written to a file, mention the exact path and whether it is committed, uncommitted, or only a local artifact.