agilab-runbook

name	agilab-runbook
description	Runbook for working in the AGILab repo (uv, Streamlit, run configs, packaging, troubleshooting).
license	BSD-3-Clause (see repo LICENSE)
metadata	{"short-description":"AGILab repo runbook","updated":"2026-05-09T00:00:00.000Z"}

AGILab runbook (Agent Skill)

Use this skill when you need repo-specific “how we do things” guidance in agilab/: launching Streamlit, regenerating run-config wrappers, debugging installs, or preparing releases.

AGILab working rules (repo policy)

• Use uv for all runs so dependencies resolve in managed envs:
  • uv --preview-features extra-build-dependencies run python …
  • uv --preview-features extra-build-dependencies run streamlit …
• No repo uvx: do not run uvx agilab from this checkout (it will run the published wheel and ignore local changes).
• Process ownership: treat existing terminals, Codex CLI sessions, dev servers, and other long-running processes as user-owned unless this turn started them. Do not use broad termination commands such as pkill, killall, pkill -f, or port-based kill pipelines that can match unrelated sessions. Stop only verified PIDs or tool sessions created for the active task. Do not use Codex CLI control shortcuts such as /stop, Esc interruption, or terminal-close actions to manage background terminals unless the terminal/session was created by this active task and its identity is verified. A status banner that says a background terminal is running is not ownership proof. If a port is busy, choose another port or ask before stopping its owner; do not try to "pause" another Codex CLI session from here.
• High-frequency shortcuts: prefer ./dev <shortcut> for repeated local validation loops. The top shortcuts are impact for impact validation, bugfix for impact plus a fast GA-selected regression run, test for targeted pytest -q, regress for GA-selected fast regression subsets, flow for one or more workflow parity profiles, badge for the explicit release/pre-release coverage-badge guard, and docs for docs mirror sync plus stamp verification. impact tells you what must be validated, test runs the narrow pytest slice, bugfix is the default low-load pre-push loop for normal code fixes, regress optimizes a likely regression subset from changed files and optional JUnit timings, flow matches local GitHub workflow profiles, badge checks badge freshness when intentionally requested, and docs keeps the public mirror aligned. Add --print-only to inspect the expanded commands.
• Run config parity: after editing .idea/runConfigurations/*.xml, regenerate wrappers:
  • uv --preview-features extra-build-dependencies run python tools/generate_runconfig_scripts.py
• PyCharm source-root switching: the JetBrains SDK named uv (agilab) is global and should point to one AGILAB source checkout at a time. To intentionally switch PyCharm execution to another checkout, run from the target checkout:
  • uv sync
  • AGILAB_PYCHARM_ALLOW_SDK_REBIND=1 uv --preview-features extra-build-dependencies run python pycharm/setup_pycharm.py Do not rerun full install.sh just to switch PyCharm roots; use it only when installer side effects are needed, such as app installation, .agilab-path updates, dataset seeding, or install-time tests.
• Source/end-user path drift: if a source Streamlit run shows end-user paths such as $HOME/agi-space/apps/builtin/<app>/.venv in the ORCHESTRATE header, treat it as a bootstrap/root-selection bug until proven otherwise. Check the active process root, PyCharm SDK binding, sys.path, ~/.local/share/agilab/.agilab-path, and persisted ~/.agilab/.env keys such as APPS_PATH and IS_SOURCE_ENV. Restart Streamlit after fixing bootstrap code because st.session_state["env"] and env.active_app can retain stale paths across reruns.
• PyCharm module hygiene: treat .idea/modules/, .idea/modules.xml, and .idea/pyProjectModel.xml as local/generated IDE state. Tracked .idea/modules/*.iml files should be removed from Git and regenerated by pycharm/setup_pycharm.py. After setup, the aligned state is:
  • no root-level .idea/*.iml
  • no duplicate .idea/**/*@*.iml or .idea/**/*#*.iml
  • modules.xml only references valid $PROJECT_DIR$/.idea/modules/*.iml files
  • .idea/modules/ exists locally but is ignored If PyCharm is open and recreates duplicate module descriptors, do not kill user-owned IDE/processes; rerun setup and remove only the stale root/numbered descriptors and their modules.xml refs.
• Local-first validation: do not jump to GitHub Actions when the same check can be run locally. Reproduce with the narrowest local command first: targeted pytest, isolated coverage commands, py_compile, Sphinx builds, or publish dry-runs. Reserve coverage badge generation for release/pre-release validation or badge tooling changes. Use CI only for GitHub-only behavior such as runner differences, OS/Python matrix coverage, permissions/secrets, or the final publish/deploy step.
• Impact triage first: for non-trivial diffs, run uv --preview-features extra-build-dependencies run python tools/impact_validate.py --staged before editing further or pushing. Use its output to decide:
  • whether the change is app-local or shared-core
  • which targeted tests are required
  • whether installer repros are mandatory
  • whether generated artifacts must be refreshed
• Diff-aware pre-push guards: keep git config core.hooksPath .githooks enabled. The hook uses tools/pre_push_changed_files.py to run docs mirror checks only for docs mirror inputs and release-proof checks only for release-proof inputs. If classification fails, it runs all local guards. Coverage badge freshness remains an explicit ./dev badge or release/pre-release check.
• Install-log startup check: before launching flows after installer changes or a reported install failure, inspect the latest installer log for errors. On macOS/Linux: dir="$HOME/log/install_logs"; f=$(ls -1t "$dir"/*.log 2>/dev/null | head -1); [ -n "$f" ] && echo "Log: $f" && grep -Eai "error|exception|traceback|failed|fatal|denied|missing|not found" "$f" | tail -n 25 || echo "No logs found." On Windows PowerShell: ($d = "$HOME\log\install_logs"); $f = Get-ChildItem -LiteralPath $d -File | Sort-Object LastWriteTime -Descending | Select-Object -First 1; if ($f) { Write-Host "Log:" $f.FullName; Select-String -LiteralPath $f.FullName -Pattern '(?i)(error|exception|traceback|failed|fatal|denied|missing|not found)' | Select-Object -Last 25 | ForEach-Object { $_.Line } } else { Write-Host "No logs found." }
• Clone policy: in the PROJECT page, keep two clone classes explicit:
  • temporary clones may share the source .venv by symlink for lightweight local experiments
  • working clones should detach .venv and rerun INSTALL before EXECUTE Do not treat a shared .venv clone as a durable environment, and do not leave renamed projects with .venv symlinks pointing at the old project path.
• Shared core approval gate: do not edit shared core technology without explicit user approval first. This includes src/agilab/core/agi-env, src/agilab/core/agi-node, src/agilab/core/agi-cluster, src/agilab/core/agi-core, shared installer/build/deploy code, and generic helpers reused across apps/pages. Prefer app-local fixes first. If a core edit looks necessary, stop and explain the required files, blast radius, and validation plan before making the change.
• Docs source of truth: edit docs in the sibling repo ../thales_agilab/docs/source relative to the active AGILAB checkout.
• Generated docs in this repo: treat docs/html (including docs/html/_sources) as build output only. Do not hand-edit files in docs/html; always edit source first and regenerate from ../thales_agilab/docs/source.
  • Canonical rebuild command: uv --preview-features extra-build-dependencies run --project ../thales_agilab --group sphinx python -m sphinx -b html ../thales_agilab/docs/source docs/html
• Streamlit API: do not add st.experimental_rerun(); use st.rerun.
• No silent fallbacks: avoid runtime “auto-fallbacks” between API clients or parameter rewrites; fail fast with actionable errors.
• Generated WORKFLOW code: keep safe-action contracts as the default assistant path for dataframe transformations. The model should return versioned JSON, AGILAB validates it against the dataframe schema, and AGILAB converts it into deterministic pandas code. Treat raw Python generation as explicit advanced mode; reserve process/container/VM sandbox guidance for raw Python execution, untrusted apps, or shared sensitive deployments, not the normal lightweight path.
• Skill placement guardrail: repo-managed skills under .claude/skills/ and .codex/skills/ must stay AGILAB-specific, cross-repo reusable for AGILAB work, or directly support this repository’s workflows. Personal skills or skills for non-AGILAB domains belong in ~/.codex/skills/ or the relevant private repo.
• Repository update requests: when the user asks to "update repos", "sync repos", or similar, first show the exact command plan before executing it. The plan should be a fenced bash block with concrete git -C <repo> commands for each targeted checkout. Use the fast path by default: status --porcelain=v1 --untracked-files=no, fetch --prune, rev-list --left-right --count HEAD...@{u}, then merge --ff-only @{u} only for repos that are actually behind. This avoids a redundant fetch from git pull and avoids slow untracked scans. Group independent repo checks and fetches in parallel when the tooling allows it. If a checkout has tracked dirty paths, do not merge it until the dirty paths are reported and the update plan is adjusted.

Git footprint maintenance

• Distinguish clearly between:
  • working-tree footprint (.venv, caches, build artifacts)
  • local Git footprint (.git/objects, .git/lfs)
  • remote repository history size
• If the user asks to reduce .git only, do not touch .venv.
• Measure before acting:
  • du -sh .git .git/objects .git/lfs
  • git count-objects -vH
  • git lfs prune --dry-run
• Prefer the safest local win first:
  • run git lfs prune when the dry-run shows meaningful reclaimable space
  • this reduces local .git/lfs without rewriting history
• For actual history reduction:
  • use git filter-repo, never ad hoc low-level object surgery
  • work in an isolated --mirror clone, not in the main checkout
  • create a backup bundle before rewriting: git bundle create /tmp/<repo>-pre-rewrite.bundle --all
  • preserve any uncommitted local files outside the checkout before realigning branches
  • rewrite only the intended refs/paths; avoid touching gh-pages or unrelated refs unless requested
  • after force-pushing rewritten refs, realign the local checkout to the new origin/* history and run:
    • git reflog expire --expire=now --all
    • git gc --prune=now
• Typical low-value history targets:
  • generated docs/html/**
  • .idea/shelf/**
  • obsolete legacy paths or duplicated archives
• Do not promise a smaller remote repository from local pruning alone. Local LFS prune and local GC only affect the clone on disk.

Common commands (from the runbook matrix)

• Impact triage:
  • cd "$PROJECT_DIR" && uv --preview-features extra-build-dependencies run python tools/impact_validate.py --staged
• Impact triage for planned paths:
  • cd "$PROJECT_DIR" && uv --preview-features extra-build-dependencies run python tools/impact_validate.py --files src/agilab/orchestrate_execute.py test/test_orchestrate_execute.py
• Dev UI: cd "$PROJECT_DIR" && uv --preview-features extra-build-dependencies run streamlit run src/agilab/main_page.py -- --openai-api-key "…" --apps-path src/agilab/apps
• Apps-pages smoke: cd "$PROJECT_DIR" && uv --preview-features extra-build-dependencies run python tools/smoke_preinit.py --active-app src/agilab/apps/builtin/flight_project --timeout 20
• Apps-pages regression (AppTest): cd "$PROJECT_DIR" && uv --preview-features extra-build-dependencies run pytest -q test/test_view_maps_network.py
• Publish dry-run (TestPyPI): cd "$PROJECT_DIR" && uv --preview-features extra-build-dependencies run python tools/pypi_publish.py --repo testpypi --dry-run --verbose
• Publish to PyPI: cd "$PROJECT_DIR" && uv --preview-features extra-build-dependencies run python tools/pypi_publish.py --repo pypi --verbose --git-tag --git-commit-version --git-reset-on-failure
  • Real PyPI publishes now require the GitHub CLI (gh) because tools/pypi_publish.py creates or updates the matching GitHub Release after pushing the tag.
  • Real PyPI does not auto-select .postN when the date version already exists. If YYYY.MM.DD is already published, choose an explicit version such as --version YYYY.MM.DD.post1; dry-run that exact command first, then publish with the same explicit version.
  • Add --delete-former-github-release only when the public release page should keep a single current GitHub Release. This deletes the previous GitHub Release entry after the new one is created, but keeps the previous git tag and PyPI files.
  • Add --delete-pypi-release <version> only when a specific old PyPI version must be removed from the selected packages. This uses an exact pypi-cleanup --version-regex match, requires real PyPI web-login credentials in [pypi_cleanup], and cannot use API tokens or trusted publishing credentials.

CI and badge checks

• Prefer local reproduction before rerunning workflows:
  • if a failing step has a local command equivalent, run that first and fix locally
  • only rerun a workflow after the local equivalent is green or when the issue is GitHub-specific
• CI badge is pinned to main:
  • https://github.com/ThalesGroup/agilab/actions/workflows/ci.yml/badge.svg?branch=main
• When checking recent workflow state, prefer the GitHub Actions runs API:
  • uv --preview-features extra-build-dependencies run python - <<'PY' ... https://api.github.com/repos/ThalesGroup/agilab/actions/workflows/ci.yml/runs?per_page=10 ... PY
• Public job logs may not be directly retrievable without auth. Use the runs/jobs API first to identify the failing step, then reproduce that exact command locally.
• For AGILAB specifically, the GitHub README now uses a static, versioned PyPI badge committed under badges/:
  • https://raw.githubusercontent.com/ThalesGroup/agilab/main/badges/pypi-version-agilab.svg
• The live PyPI page can still lag until a new package is actually published; do not infer package publication from the GitHub badge alone.
• After a release, verify all four surfaces separately before trusting version state:
  • PyPI JSON: https://pypi.org/pypi/agilab/json
  • PyPI simple index: https://pypi.org/simple/agilab/
  • GitHub Release: gh release list --limit 5 and gh release view <tag>
  • GitHub static badge: https://raw.githubusercontent.com/ThalesGroup/agilab/main/badges/pypi-version-agilab.svg
• Verify the published wheel from outside the repo checkout so imports cannot resolve to local source:
  • cd /tmp && uv run --refresh-package agilab --no-project --with agilab==<version> python -c "import importlib.metadata as m; print(m.version('agilab'))"
• If the release is followed by a Hugging Face deploy, refresh release proof metadata with the live Space commit after runtime cutover, then sync and push both docs repos:
  • uv --preview-features extra-build-dependencies run python tools/release_proof_report.py --docs-source ../thales_agilab/docs/source --refresh-from-local --hf-space-commit <space-sha> --render --check --compact
  • uv --preview-features extra-build-dependencies run python tools/sync_docs_source.py --source ../thales_agilab/docs/source --target docs/source --apply --delete
• If the version changes, update the static badge and GitHub Release in the same commit series as the version bump so main, PyPI, the README, and release metadata stay aligned.

CI workflow lessons

• The root src/agilab/test step is more stable when run from the source tree instead of the full project environment:
  • PYTHONPATH='src' COVERAGE_FILE=.coverage.agilab uv --preview-features extra-build-dependencies run --no-project --with pytest --with pytest-cov --with toml --with packaging python -m pytest ... --ignore=src/agilab/test/test_model_returns_code.py src/agilab/test
• The integration-only src/agilab/test/test_model_returns_code.py should be ignored in CI collection, not merely deselected by marker, because import-time behavior can still break collection.
• Core package coverage steps are more reliable when each step uses an isolated no-project env with explicit editable core packages and test-only extras, instead of relying on the monorepo root env.
• agi-env tests need:
  • editable ./src/agilab/core/agi-env
  • editable ./src/agilab/core/agi-node
  • sqlalchemy
• Shared core tests (src/agilab/core/test) need:
  • editable ./src/agilab/core/agi-env
  • editable ./src/agilab/core/agi-node
  • editable ./src/agilab/core/agi-cluster
  • editable ./src/agilab/core/agi-core
  • sqlalchemy
  • pytest-asyncio
• Coverage combine/XML generation should use an isolated coverage toolchain too:
  • uv --preview-features extra-build-dependencies run --no-project --with coverage --with pytest-cov python -m coverage ...

Troubleshooting reminders

• Source run unexpectedly points to $HOME/agi-space:
  • confirm the launched script path is the current checkout, not the packaged install
  • confirm the PyCharm SDK was rebound to the intended checkout when switching roots
  • inspect ~/.local/share/agilab/.agilab-path and ~/.agilab/.env before trusting UI headers
  • reproduce with a cold Streamlit/AppTest session; a warm session can keep a stale env object even after the source code is fixed
• Missing import: check both manager and worker pyproject.toml scopes (src/agilab/apps/<app>/pyproject.toml and src/agilab/apps/<app>/src/<app>_worker/pyproject.toml).
• Installer pip issue: run uv --preview-features extra-build-dependencies run python -m ensurepip --upgrade once in the target venv.
• Cluster inventory/status mismatch:
  • If the UI shows a worker as unreachable but ssh <user>@<ip> 'echo ok' works, reproduce the exact non-interactive probe path used by AGILAB before changing UI display code.
  • Check remote PATH and required tools with SSH, not an interactive shell: ssh <user>@<ip> 'printf "path=%s\n" "$PATH"; command -v python3; command -v nvidia-smi || true; uname -a'.
  • Validate the same account can reach the scheduler and shared storage from the worker: ssh <user>@<ip> 'ssh -o BatchMode=yes <scheduler_user>@<scheduler_ip> hostname' and the configured cluster-share mount/read-write sentinel.
  • Treat a display of "+ 1 worker unreachable" as an inventory/probe failure until the exact probe command succeeds; a bare SSH success only proves authentication.
• For a reinstalled cluster node, separate host-key repair from auth repair:
  • host key changed:
    • ssh-keygen -R <ip>
    • ssh-keyscan -H -t ed25519 <ip> >> ~/.ssh/known_hosts
  • user key missing on remote:
    • ssh-copy-id agi@<ip>
    • or recreate ~/.ssh/authorized_keys with 0700 / 0600 permissions
• If cluster mode depends on shared storage, restore the node’s .agilab/.env and remount the share before blaming AGILAB:
  • Linux node example:
    • AGI_CLUSTER_SHARE=/home/<worker-user>/clustershare
    • AGI_LOCAL_SHARE=/home/<worker-user>/localshare
    • sshfs <manager-user>@<manager-ip>:/path/to/manager/clustershare /home/<worker-user>/clustershare
• For macOS SSHFS workers:
  • command -v brew can miss Intel Homebrew; check /usr/local/Homebrew/bin/brew before assuming Homebrew is absent.
  • Prefer an interactive package install of FUSE-T SSHFS or macFUSE plus SSHFS; the package step may need an admin password.
  • Confirm non-interactive SSH can find sshfs: ssh <user>@<worker> 'command -v sshfs'.
  • If sshfs is under /usr/local/bin but not found over SSH, add /usr/local/bin in the remote user’s ~/.zshenv.
  • Validate reverse SSH too: ssh <user>@<worker> 'ssh -o BatchMode=yes <manager_user>@<manager_ip> hostname'.
  • Fix worker-side known_hosts first, then add the worker public key to the manager account if reverse auth fails.
• After a reinstall, validate both directions explicitly before rerunning installs:
  • ssh agi@<ip> 'echo ok'
  • ssh agi@<ip> 'ssh -o BatchMode=yes agi@<scheduler_ip> hostname'