agilab-runbook

name	agilab-runbook
description	Runbook for working in the AGILab repo (uv, Streamlit, run configs, packaging, troubleshooting).
license	BSD-3-Clause (see repo LICENSE)
metadata	{"short-description":"AGILab repo runbook","updated":"2026-06-19T00:00:00.000Z"}

AGILab runbook (Agent Skill)

Use this skill when you need repo-specific “how we do things” guidance in agilab/: launching Streamlit, regenerating run-config wrappers, debugging installs, or preparing releases.

AGILab working rules (repo policy)

• Use uv for all runs so dependencies resolve in managed envs:
  • uv --preview-features extra-build-dependencies run python …
  • uv --preview-features extra-build-dependencies run --extra ui streamlit …
• No repo uvx: do not run uvx agilab from this checkout (it will run the published wheel and ignore local changes).
• Process ownership: treat existing terminals, Codex CLI sessions, dev servers, and other long-running processes as user-owned unless this turn started them. Do not use broad termination commands such as pkill, killall, pkill -f, or port-based kill pipelines that can match unrelated sessions. Stop only verified PIDs or tool sessions created for the active task. Do not use Codex CLI control shortcuts such as /stop, Esc interruption, or terminal-close actions to manage background terminals unless the terminal/session was created by this active task and its identity is verified. A status banner that says a background terminal is running is not ownership proof. If a port is busy, choose another port or ask before stopping its owner; do not try to "pause" another Codex CLI session from here.
• Tokki terminal routing: when Tokki is available and the command can be executed faithfully, run ad-hoc terminal checks through tokki run -- <command> so noisy output is digested and full logs stay in private artifacts. Use raw commands instead when a tool needs live streaming, TTY interaction, structured stdout, or exact unwrapped behavior.
• High-frequency shortcuts: prefer ./dev <shortcut> for repeated local validation loops. The top shortcuts are impact for impact validation, bugfix for impact plus a fast GA-selected regression run, test for targeted pytest -q, regress for GA-selected fast regression subsets, builtin-app-tests for app-local built-in app pytest suites, flow for one or more workflow parity profiles, badge for the explicit release/pre-release coverage-badge guard, and docs for docs mirror sync plus stamp verification. impact tells you what must be validated, test runs the narrow pytest slice, bugfix is the default low-load pre-push loop for normal code fixes, regress optimizes a likely regression subset from changed files and optional JUnit timings, builtin-app-tests runs each built-in app's tests inside that app's own uv --project . environment with pytest importlib collection, flow matches local GitHub workflow profiles, badge checks badge freshness when intentionally requested, and docs keeps the public mirror aligned. Add --print-only to inspect the expanded commands.
• Run config parity: after editing .idea/runConfigurations/*.xml, regenerate wrappers:
  • uv --preview-features extra-build-dependencies run python tools/generate_runconfig_scripts.py
• PyCharm source-root switching: the JetBrains SDK named uv (agilab) is global and should point to one AGILAB source checkout at a time. To intentionally switch PyCharm execution to another checkout, run from the target checkout:
  • uv sync
  • AGILAB_PYCHARM_ALLOW_SDK_REBIND=1 uv --preview-features extra-build-dependencies run python pycharm/setup_pycharm.py Do not rerun full install.sh just to switch PyCharm roots; use it only when installer side effects are needed, such as app installation, .agilab-path updates, dataset seeding, or install-time tests.
• Source/end-user path drift: if a source Streamlit run shows end-user paths such as $HOME/agi-space/apps/builtin/<app>/.venv in the ORCHESTRATE header, treat it as a bootstrap/root-selection bug until proven otherwise. Check the active process root, PyCharm SDK binding, sys.path, ~/.local/share/agilab/.agilab-path, and persisted ~/.agilab/.env keys such as APPS_PATH and IS_SOURCE_ENV. Restart Streamlit after fixing bootstrap code because st.session_state["env"] and env.active_app can retain stale paths across reruns.
• Worker data path resolution layer: in cluster and workflow execution, UI pages may choose and persist workers_data_path, but canonical share-root storage and app data_in / data_out resolution belong to the shared worker runtime: runtime_misc_support.initialize_runtime_state, BaseWorker._resolve_data_dir, and agi_node.agi_dispatcher.base_worker_path_support. The canonical workflow share root passed as workers_data_path is clustershare/<user>/<workflow-id>/<session>. App module subdirectories are appended by app arguments, where the module is the project name without the _project suffix, yielding clustershare/<user>/<workflow-id>/<session>/<module>/... for app data. Do not fix duplicated paths such as <module>/<module>/dataset, stale <project>/<session>/workers, or UI value re-aggregation in one app page only. Fix the shared resolver when the failure class is generic, keep UI fixes limited to default selection/persistence, and add regressions at both test_base_worker_path_support.py and test_base_worker.py plus a focused UI test only when the default/persisted value changes. Do not preserve backward compatibility for the legacy /workers layout unless an explicit migration request requires it.
• Source Streamlit launches must be non-mutating by default: if starting agilab run (dev) prints Uninstalled ... or Installed ..., inspect both the outer run-config command and any launcher-spawned inner uv run command. The selected default app is not the cause; uv is reconciling the project environment or the ui extra. Runtime launchers should preserve UV_NO_SYNC and use uv run --no-sync for the inner Streamlit command. Do dependency bootstrap explicitly with uv sync --extra ui or a clearly named sync flag before launch, not silently during normal UI startup.
• PyPI app management: promoted public apps can be managed through the PROJECT page Install PyPI app flow or agilab app search/check/install/list/update/remove. This route installs agi-app-* packages into the selected Python environment and discovers apps from agilab.apps entry points. It is separate from APPS_REPOSITORY, which remains the source-checkout route for editable app repositories.
• Source app install/test is script-first: when the source checkout is already installed and the task is only to install or test apps from APPS_REPOSITORY, run the app installer directly from src/agilab. Use APPS_REPOSITORY=/path/to/apps-repo AGILAB_DEV_APPS_REPOSITORY=1 BUILTIN_APPS=__AGILAB_ALL_APPS__ ./install_apps.sh for all installable source apps and add --test-apps for app tests. Use a comma-separated BUILTIN_APPS value for a subset, even for repository apps, because this is the script's model-free selection contract. On Windows, use the matching install_apps.ps1 -TestApps command with the same environment variables. Do not replace this with a root reinstall, hand-edited ~/.agilab/.env, or manual pytest loops.
• PyCharm module hygiene: treat .idea/modules/, .idea/modules.xml, and .idea/pyProjectModel.xml as local/generated IDE state. Tracked .idea/modules/*.iml files should be removed from Git and regenerated by pycharm/setup_pycharm.py. After setup, the aligned state is:
  • no root-level .idea/*.iml
  • no duplicate .idea/**/*@*.iml or .idea/**/*#*.iml
  • modules.xml only references valid $PROJECT_DIR$/.idea/modules/*.iml files
  • .idea/modules/ exists locally but is ignored If PyCharm is open and recreates duplicate module descriptors, do not kill user-owned IDE/processes; rerun setup and remove only the stale root/numbered descriptors and their modules.xml refs.
• Local-first validation: do not jump to GitHub Actions when the same check can be run locally. Reproduce with the narrowest local command first: targeted pytest, isolated coverage commands, py_compile, Sphinx builds, or publish dry-runs. Reserve coverage badge generation for release/pre-release validation or badge tooling changes. Use CI only for GitHub-only behavior such as runner differences, OS/Python matrix coverage, permissions/secrets, or the final publish/deploy step.
• Current-code planning guardrail: before answering "next move", "ready for release", "release it", "sync HF", or any operational sequencing question, inspect the current repo state and authoritative workflow/tooling files. For release sequencing, check ./dev --print-only release, .github/workflows/pypi-publish.yaml, and tools/release_plan.py before saying whether PyPI, GitHub release assets, Hugging Face sync, release proof, or docs updates are separate manual steps. If the workflow already performs a step, state its condition instead of adding a redundant manual follow-up.
• Impact triage first: for non-trivial diffs, run uv --preview-features extra-build-dependencies run python tools/impact_validate.py --staged before editing further or pushing. Use its output to decide:
  • whether the change is app-local or shared-core
  • which targeted tests are required
  • whether installer repros are mandatory
  • whether generated artifacts must be refreshed
• Diff-aware pre-push guards: keep git config core.hooksPath .githooks enabled. The hook uses tools/pre_push_changed_files.py to run docs mirror checks only for docs mirror inputs, release-proof checks only for release-proof inputs, and agent-instruction contract checks only for AGENTS/runbook/skill/discovery inputs. If classification fails, it runs all local guards. Coverage badge freshness remains an explicit ./dev badge or release/pre-release check.
• Public style badge alignment: keep the code-style/style-guard signal in README.md and README.pypi.md aligned with the actual repository guard. If the public badge moves to a Ruff changed-files guard or another style tool, update both README files, grep for stale Black, black, or code%20style badge text, and avoid advertising a tool that is not wired into the local validation path.
• Install-log startup check: before launching flows after installer changes or a reported install failure, inspect the latest installer log for errors. On macOS/Linux: dir="$HOME/log/install_logs"; f=$(ls -1t "$dir"/*.log 2>/dev/null | head -1); [ -n "$f" ] && echo "Log: $f" && grep -Eai "error|exception|traceback|failed|fatal|denied|missing|not found" "$f" | tail -n 25 || echo "No logs found." On Windows PowerShell: ($d = "$HOME\log\install_logs"); $f = Get-ChildItem -LiteralPath $d -File | Sort-Object LastWriteTime -Descending | Select-Object -First 1; if ($f) { Write-Host "Log:" $f.FullName; Select-String -LiteralPath $f.FullName -Pattern '(?i)(error|exception|traceback|failed|fatal|denied|missing|not found)' | Select-Object -Last 25 | ForEach-Object { $_.Line } } else { Write-Host "No logs found." }
• Windows compatibility contract: distinguish native Windows package support from source-checkout convenience. The released package CLI first-proof and repo-guardrails clean install matrix are expected to work on Windows. The source checkout installer still includes POSIX shell paths, so prefer WSL2 for that path unless the task explicitly targets install.ps1 or native Windows parity. Do not remove the Windows classifier only because a source install.sh path needs WSL2.
• Windows process and path handling: when touching Streamlit sidecars or app/page launch helpers, prefer argv-list commands with shell=False; avoid POSIX-only shlex.quote strings for commands that may run on Windows. For clone/app-repository links, allow Windows directory junction fallback when symlink creation is denied, and treat missing link privileges as a recoverable condition where possible.
• Windows network shares: never hard-code placeholder net use credentials. BaseWorker should attempt network-drive mapping only when AGILAB_WINDOWS_NET_USE_USER and AGILAB_WINDOWS_NET_USE_PASSWORD are set, with AGILAB_WINDOWS_NET_USE_DRIVE optionally selecting the drive letter. Missing credentials should skip mapping with an informational log.
• Clone policy: in the PROJECT page, keep two clone classes explicit:
  • temporary clones may share the source .venv by symlink for lightweight local experiments
  • working clones should detach .venv and rerun INSTALL before EXECUTE Do not treat a shared .venv clone as a durable environment, and do not leave renamed projects with .venv symlinks pointing at the old project path.
• Shared core approval gate: do not edit shared core technology without explicit user approval first. This includes src/agilab/core/agi-env, src/agilab/core/agi-node, src/agilab/core/agi-cluster, src/agilab/core/agi-core, shared installer/build/deploy code, and generic helpers reused across apps/pages. Prefer app-local fixes first. If a core edit looks necessary, stop and explain the required files, blast radius, and validation plan before making the change.
• Higher-model fix review: when a product or code fix was designed or implemented with model assistance, request a review from a stronger model before closing, pushing, or merging when that is available. The review should challenge root cause, regression chain, blast radius, security risk, and test coverage. If no stronger model is available in the current environment, state that explicitly and still perform the normal local review and validation path.
• Docs source of truth: edit docs in the sibling repo ../thales_agilab/docs/source relative to the active AGILAB checkout.
• Generated docs in this repo: treat docs/html (including docs/html/_sources) as build output only. Do not hand-edit files in docs/html; always edit source first and regenerate from ../thales_agilab/docs/source.
  • Canonical rebuild command: uv --preview-features extra-build-dependencies run --project ../thales_agilab --group sphinx python -m sphinx -b html ../thales_agilab/docs/source docs/html
• Streamlit API: do not add st.experimental_rerun(); use st.rerun.
• No silent fallbacks: avoid runtime “auto-fallbacks” between API clients or parameter rewrites; fail fast with actionable errors.
• Generated WORKFLOW code: keep safe-action contracts as the default assistant path for dataframe transformations. The model should return versioned JSON, AGILAB validates it against the dataframe schema, and AGILAB converts it into deterministic pandas code. Treat raw Python generation as explicit advanced mode; reserve process/container/VM sandbox guidance for raw Python execution, untrusted apps, or shared sensitive deployments, not the normal lightweight path.
• Worker artifact evidence: do not create a new worker subclass just to record outputs. BaseWorker exposes the shared artifact contract, so DagWorker, PandasWorker, PolarsWorker, and app workers can call record_artifact(...), record_metric(...), and write_manifest(...) to produce standard worker evidence without adding dataframe or UI dependencies.
• Skill placement guardrail: repo-managed skills under .claude/skills/ and .codex/skills/ must stay AGILAB-specific, cross-repo reusable for AGILAB work, or directly support this repository’s workflows. Personal skills or skills for non-AGILAB domains belong in ~/.codex/skills/ or the relevant private repo.
• Repository update requests: when the user asks to "update repos", "sync repos", or similar, first show the exact command plan before executing it. The plan should be a fenced bash block with concrete git -C <repo> commands for each targeted checkout. In AGILAB maintenance, the default target set is the active agilab checkout plus the sibling ../thales_agilab canonical docs/apps checkout when it exists; include both in the printed plan or explicitly say why one is missing, unsafe, or out of scope. Use the fast path by default: status --porcelain=v1 --untracked-files=no, fetch --prune, rev-list --left-right --count HEAD...@{u}, then merge --ff-only @{u} only for repos that are actually behind. This avoids a redundant fetch from git pull and avoids slow untracked scans. Group independent repo checks and fetches in parallel when the tooling allows it. If a checkout has tracked dirty paths, do not merge it until the dirty paths are reported and the update plan is adjusted. If a dirty feature checkout blocks direct update of a repo's main, check whether a clean registered main worktree exists before omitting that repository from the update.
• PR agent metadata: every AGILAB PR description must include an Agent Metadata section with the Tokki version (tokki --version, or not used/unavailable), agent/runtime name and version when exposed, model name, reasoning effort, and whether /fast mode was used. Do not infer missing values; write unknown, unavailable, or not used explicitly.
• PR evidence contract: PR descriptions must stay current through merge. Include Review Evidence when model or sub-agent review was used, naming the reviewer model, result, and whether findings were addressed. If sub-agents were used, state their role/model and whether they edited files or reviewed only. Bugfix PRs must include Repro, Root Cause, and Regression Test sections; if automation is impractical, say why and name the closest validation. Any skipped check must include a reason such as not applicable, GitHub skipped, or manual validation only. Before marking a PR ready or merging, update the body if review, CI, validation, or skip status changed after PR creation.
• External AGILAB core pointer merges: when a private app or integration repository points .external/agilab at a stale AGILAB core branch and GitHub reports that the branch has no history in common with current main, do not force-merge or preserve the stale branch by default. Fetch the public AGILAB remote, check whether the intended fix already exists on origin/main, move the external pointer to the current public origin/main when it does, rerun the full private app-local validation against that pointer, then commit only the pointer update in the integration repository. After the pointer commit is pushed and validated, delete the stale unmergeable public branch.
• Dirty worktree cleanup: when cleaning stale local worktrees, do not delete dirty worktrees blindly. First inspect git -C <worktree> status --short and the branch relationship to origin/main. If dirty changes are obsolete but still worth preserving, archive git -C <worktree> diff --binary plus a status snapshot outside the repo before removal. Remove only clean worktrees or dirty worktrees with an explicit archive path, then run git worktree prune, delete stale branch refs that are already represented on origin/main, and realign local main to origin/main when the old main worktree has been removed safely.

Git footprint maintenance

• Distinguish clearly between:
  • working-tree footprint (.venv, caches, build artifacts)
  • local Git footprint (.git/objects, .git/lfs)
  • remote repository history size
• If the user asks to reduce .git only, do not touch .venv.
• Measure before acting:
  • du -sh .git .git/objects .git/lfs
  • git count-objects -vH
  • git lfs prune --dry-run
• Prefer the safest local win first:
  • run git lfs prune when the dry-run shows meaningful reclaimable space
  • this reduces local .git/lfs without rewriting history
• For actual history reduction:
  • use git filter-repo, never ad hoc low-level object surgery
  • work in an isolated --mirror clone, not in the main checkout
  • create a backup bundle before rewriting: git bundle create /tmp/<repo>-pre-rewrite.bundle --all
  • preserve any uncommitted local files outside the checkout before realigning branches
  • rewrite only the intended refs/paths; avoid touching gh-pages or unrelated refs unless requested
  • after force-pushing rewritten refs, realign the local checkout to the new origin/* history and run:
    • git reflog expire --expire=now --all
    • git gc --prune=now
• Typical low-value history targets:
  • generated docs/html/**
  • .idea/shelf/**
  • obsolete legacy paths or duplicated archives
• Do not promise a smaller remote repository from local pruning alone. Local LFS prune and local GC only affect the clone on disk.

Common commands (from the runbook matrix)

• Impact triage:
  • cd "$PROJECT_DIR" && uv --preview-features extra-build-dependencies run python tools/impact_validate.py --staged
• Impact triage for planned paths:
  • cd "$PROJECT_DIR" && uv --preview-features extra-build-dependencies run python tools/impact_validate.py --files src/agilab/orchestrate_execute.py test/test_orchestrate_execute.py
• Dev UI: cd "$PROJECT_DIR" && uv --preview-features extra-build-dependencies run --extra ui streamlit run src/agilab/main_page.py -- --openai-api-key "…" --apps-path src/agilab/apps
• Apps-pages smoke: cd "$PROJECT_DIR" && uv --preview-features extra-build-dependencies run python tools/smoke_preinit.py --active-app src/agilab/apps/builtin/flight_project --timeout 20
• Apps-pages regression (AppTest): cd "$PROJECT_DIR" && uv --preview-features extra-build-dependencies run pytest -q test/test_view_maps_network.py
• First-adoption handoff:
  • run agilab first-proof --json --with-ui or the equivalent source command to create ~/log/execute/flight_telemetry/run_manifest.json
  • run agilab adoption-report for the human checkpoint, or agilab adoption-report --json --strict for automation
  • treat safe_to_expand=true as permission to try the next demo/notebook lane, not as production, shared-workstation, secrets, public-exposure, or cluster approval
  • for team handoff, keep the manifest, exported agi-core notebooks/lab_stages.ipynb, compatibility-report output, and redacted strict security-check output together. The notebook handoff removes dependence on the AGILAB UI/distributed-worker layer, but it still relies on the stable core runtime and the exported project's dependencies.
• PyPI app-package management:
  • search: agilab app search flight
  • preflight: agilab app check agi-app-flight-telemetry --json
  • install: agilab app install agi-app-flight-telemetry
  • inventory/update/remove: agilab app list, agilab app update <package>, agilab app remove <package>
• Publish dry-run (TestPyPI): cd "$PROJECT_DIR" && uv --preview-features extra-build-dependencies run python tools/pypi_publish.py --repo testpypi --dry-run --verbose
• Publish to PyPI: cd "$PROJECT_DIR" && uv --preview-features extra-build-dependencies run python tools/pypi_publish.py --repo pypi --verbose --git-tag --git-commit-version --git-reset-on-failure
  • Real PyPI publishes now require the GitHub CLI (gh) because tools/pypi_publish.py creates or updates the matching GitHub Release after pushing the tag.
  • Real PyPI does not auto-select .postN when the date version already exists. Normal public releases should move to a deliberate new date-based version or a release candidate/TestPyPI rehearsal first. Public .postN releases are forbidden for new AGILAB publications; use release_mode=hotfix with YYYY.MM.DD.N for a same-day public fix.
  • On publish retries, the top-level agilab artifact may already exist while split runtime/app/page packages still need publication. Let the preflight/release plan decide what remains instead of checking only the root package.
  • Add --delete-former-github-release only when the public release page should keep a single current GitHub Release. This deletes the previous GitHub Release entry after the new one is created, but keeps the previous git tag and PyPI files.
  • Add --delete-pypi-release <version> only when a specific old PyPI version must be removed from the selected packages. This uses an exact pypi-cleanup --version-regex match, requires real PyPI web-login credentials in [pypi_cleanup], and cannot use API tokens or trusted publishing credentials.
  • PyPI "confirmed device" state is browser/account state, not a CLI or macOS-wide setting. For web cleanup flows, log into https://pypi.org/ from a normal browser profile on the maintainer machine, keep PyPI cookies/site data, and add the Mac Touch ID/passkey under PyPI account two-factor security-device settings if needed. Do not commit passwords, recovery codes, cookies, browser profiles, or PyPI session material.

CI and badge checks

• Prefer local reproduction before rerunning workflows:
  • if a failing step has a local command equivalent, run that first and fix locally
  • only rerun a workflow after the local equivalent is green or when the issue is GitHub-specific
• If a workflow is cancelled, fetch and compare the cancelled run head SHA with current origin/main before debugging. A newer push often supersedes coverage on main; follow the current head's run instead of fixing a cancelled predecessor unless the same failure reproduces on the latest head.
• CI badge is pinned to main:
  • https://github.com/ThalesGroup/agilab/actions/workflows/ci.yml/badge.svg?branch=main
• When checking recent workflow state, prefer the GitHub Actions runs API:
  • uv --preview-features extra-build-dependencies run python - <<'PY' ... https://api.github.com/repos/ThalesGroup/agilab/actions/workflows/ci.yml/runs?per_page=10 ... PY
• Public job logs may not be directly retrievable without auth. Use the runs/jobs API first to identify the failing step, then reproduce that exact command locally.
• For AGILAB specifically, the GitHub README now uses a static, versioned PyPI badge committed under badges/:
  • https://raw.githubusercontent.com/ThalesGroup/agilab/main/badges/pypi-version-agilab.svg
• The live PyPI page can still lag until a new package is actually published; do not infer package publication from the GitHub badge alone.
• After a release, verify all four surfaces separately before trusting version state:
  • PyPI JSON: https://pypi.org/pypi/agilab/json
  • PyPI simple index: https://pypi.org/simple/agilab/
  • GitHub Release: gh release list --limit 5 and gh release view <tag>
  • GitHub static badge: https://raw.githubusercontent.com/ThalesGroup/agilab/main/badges/pypi-version-agilab.svg
  • If PyPI JSON is current but clean installs still resolve an old version, check the Simple API and use uv --refresh-package agilab or a fresh cache before declaring the publish broken.
• Also verify the GitHub deployment environments, not only the workflow conclusion:
  • the split-package publisher now publishes runtime components, UI components, page bundles, app-project payloads, the agi-pages / agi-apps umbrellas, and the top-level agilab package. Do not check only pypi-agi-apps or pypi-agi-pages; a green umbrella publish can still hide a missing payload such as pypi-agi-app-flight-project.
  • the legacy /deployments/pypi page can stay red from older releases because the current Trusted Publisher/OIDC claim intentionally no longer uses the generic pypi GitHub environment
  • derive the authoritative package/environment list from the release plan: uv --preview-features extra-build-dependencies run python tools/release_plan.py --check-workflow .github/workflows/pypi-publish.yaml
  • for a focused deployment-status check, query the environment names emitted by the release plan instead of maintaining a separate hard-coded list
  • before retiring a stale generic pypi deployment, confirm it has no rules, secrets, or variables: gh api repos/ThalesGroup/agilab/environments/pypi, gh secret list --repo ThalesGroup/agilab --env pypi, and gh variable list --repo ThalesGroup/agilab --env pypi
  • if the generic pypi environment is only stale legacy status, mark its latest failed deployment inactive and point the status at the successful pypi-publish run; do not rewrite a genuinely failed deployment as success
• Verify the published wheel from outside the repo checkout so imports cannot resolve to local source:
  • cd /tmp && uv run --refresh-package agilab --no-project --with agilab==<version> python -c "import importlib.metadata as m; print(m.version('agilab'))"
  • cd /tmp && uv --preview-features extra-build-dependencies run --refresh-package agilab --no-project --with 'agilab[examples]==<version>' python -m agilab.lab_run first-proof --json --max-seconds 60
  • Use agilab[examples] for packaged first-proof smoke; the base agilab wheel is intentionally lean and may not install demo payload dependencies.
• The public PyPI release workflow includes the sync-hf-space job after GitHub release assets. It deploys the Hugging Face Space, runs the hosted smoke test, and records the Space commit in release proof when needs.release-plan.outputs.pypi_publish_selected == 'true' and needs.publish-release-assets.result == 'success'. Do not add a separate manual HF sync step to a release plan unless that job failed, was skipped by selected package scope, or the user explicitly asks for an out-of-band Space redeploy.
• If the version changes, update the static badge and GitHub Release in the same commit series as the version bump so main, PyPI, the README, and release metadata stay aligned.

CI workflow lessons

• The root src/agilab/test step is more stable when run from the source tree instead of the full project environment:
  • PYTHONPATH='src' COVERAGE_FILE=.coverage.agilab uv --preview-features extra-build-dependencies run --no-project --with pytest --with pytest-cov --with toml --with packaging python -m pytest ... --ignore=src/agilab/test/test_model_returns_code.py src/agilab/test
• The integration-only src/agilab/test/test_model_returns_code.py should be ignored in CI collection, not merely deselected by marker, because import-time behavior can still break collection.
• Core package coverage steps are more reliable when each step uses an isolated no-project env with explicit editable core packages and test-only extras, instead of relying on the monorepo root env.
• agi-env tests need:
  • editable ./src/agilab/core/agi-env
  • editable ./src/agilab/core/agi-node
  • sqlalchemy
• Shared core tests (src/agilab/core/test) need:
  • editable ./src/agilab/core/agi-env
  • editable ./src/agilab/core/agi-node
  • editable ./src/agilab/core/agi-cluster
  • editable ./src/agilab/core/agi-core
  • sqlalchemy
  • pytest-asyncio
• Coverage combine/XML generation should use an isolated coverage toolchain too:
  • uv --preview-features extra-build-dependencies run --no-project --with coverage --with pytest-cov python -m coverage ...
• Internal package exact-pin tests must account for valid environment markers. If an app package raises its Python floor, keep the umbrella dependency marker such as ; python_version >= '3.13' instead of downgrading the app or broadening unsupported Python versions. Tests should parse requirements or compare the unmarked exact pin plus marker validity, not raw strings only.

Troubleshooting reminders

• Source run unexpectedly points to $HOME/agi-space:
  • confirm the launched script path is the current checkout, not the packaged install
  • confirm the PyCharm SDK was rebound to the intended checkout when switching roots
  • inspect ~/.local/share/agilab/.agilab-path and ~/.agilab/.env before trusting UI headers
  • reproduce with a cold Streamlit/AppTest session; a warm session can keep a stale env object even after the source code is fixed
• Missing import: check both manager and worker pyproject.toml scopes (src/agilab/apps/<app>/pyproject.toml and src/agilab/apps/<app>/src/<app>_worker/pyproject.toml).
• Installer pip issue: run uv --preview-features extra-build-dependencies run python -m ensurepip --upgrade once in the target venv.
• Cluster inventory/status mismatch:
  • Start with fresh discovery, not remembered LAN addresses: uv --preview-features extra-build-dependencies run --no-sync python tools/cluster_flight_validation.py --discover-lan --remote-user <user> --json --no-discovery-cache. Treat nodes with status="no-ssh-port" or missing ARP as stale candidates; do not use them for validation until a fresh SSH probe succeeds.
  • If the UI shows a worker as unreachable but ssh <user>@<ip> 'echo ok' works, reproduce the exact non-interactive probe path used by AGILAB before changing UI display code.
  • Check remote PATH and required tools with SSH, not an interactive shell: ssh <user>@<ip> 'printf "path=%s\n" "$PATH"; command -v python3; command -v nvidia-smi || true; uname -a'.
  • Validate the same account can reach the scheduler and shared storage from the worker: ssh <user>@<ip> 'ssh -o BatchMode=yes <scheduler_user>@<scheduler_ip> hostname' and the configured cluster-share mount/read-write sentinel.
  • Treat a display of "+ 1 worker unreachable" as an inventory/probe failure until the exact probe command succeeds; a bare SSH success only proves authentication.
• For a reinstalled cluster node, separate host-key repair from auth repair:
  • rediscover the worker IP first; examples must use <ip> placeholders, not LAN addresses remembered from an earlier session
  • host key changed:
    • ssh-keygen -R <ip>
    • ssh-keyscan -H -t ed25519 <ip> >> ~/.ssh/known_hosts
  • user key missing on remote:
    • ssh-copy-id agi@<ip>
    • or recreate ~/.ssh/authorized_keys with 0700 / 0600 permissions
• If cluster mode depends on shared storage, restore the node’s .agilab/.env and remount the share before blaming AGILAB:
  • Linux node example:
    • worker_home="<worker-home>"
    • AGI_CLUSTER_SHARE="$worker_home/clustershare"
    • AGI_LOCAL_SHARE="$worker_home/localshare"
    • sshfs <manager-user>@<manager-ip>:/path/to/manager/clustershare "$worker_home/clustershare"
• For macOS SSHFS workers:
  • command -v brew can miss Intel Homebrew; check /usr/local/Homebrew/bin/brew before assuming Homebrew is absent.
  • Prefer an interactive package install of FUSE-T SSHFS or macFUSE plus SSHFS; the package step may need an admin password.
  • Confirm non-interactive SSH can find sshfs: ssh <user>@<worker> 'command -v sshfs'.
  • If sshfs is under /usr/local/bin but not found over SSH, add /usr/local/bin in the remote user’s ~/.zshenv.
  • Validate reverse SSH too: ssh <user>@<worker> 'ssh -o BatchMode=yes <manager_user>@<manager_ip> hostname'.
  • Fix worker-side known_hosts first, then add the worker public key to the manager account if reverse auth fails.
• After a reinstall, validate both directions explicitly before rerunning installs:
  • ssh agi@<ip> 'echo ok'
  • ssh agi@<ip> 'ssh -o BatchMode=yes agi@<scheduler_ip> hostname'