antithesis-moog

name

antithesis-moog

description

MOOG CLI for Antithesis test management on Cardano. Setup using the release binary (never moog-head — they encode types differently), identity switching (requester/agent/oracle wallets via being_*), token state inspection, retracting stuck requests during an oracle crash-loop, cleaning stale "running" tests, how the agent reconciles on-chain test-runs against the Antithesis REST API (email is gone — incomplete maps to failure), and the multi-tenant Antithesis configuration introduced by https://github.com/cardano-foundation/moog/pull/93. Triggers — "moog", "moog-head", "MOOG CLI", "being_requester", "being_agent", "being_oracle", "moog token", "moog facts test-runs", "moog retract", "moog agent report-test", "moog wallet info", "UnknownUpdateValidationFailure", "oracle crash-loop", "stale running test", "Antithesis report URL", "incomplete maps to failure", "Antithesis API reconciliation", "amaru-cardano", "pragma tenant", "MOOG_ANTITHESIS_LAUNCH_URL", "MOOG_REGISTRY".

MOOG Operations

Setup — CRITICAL

Always use the release binary, never moog-head.

cd /code/moog
# Download the release binary (same as the GitHub Actions workflow uses).
# Latest release is v0.5.1.5; asset naming changed at v0.5.x — the portable,
# statically-linked CLI is now `moog-<ver>-x86_64-linux-musl.tar.gz` (it
# extracts a single `moog` binary; moog-agent / moog-oracle are separate assets).
curl -sL "https://github.com/cardano-foundation/moog/releases/download/v0.5.1.5/moog-0.5.1.5-x86_64-linux-musl.tar.gz" | tar xz -C ./tmp/
export PATH=/code/moog/tmp:$PATH
source tmp/prod-setup.sh

Asset naming history: ≤ v0.4.1.2 used moog-vX.Y.Z.W-linux64.tar.gz; v0.5.x switched to per-arch names — moog-X.Y.Z.W-x86_64-linux-musl.tar.gz (musl = static/portable), plus .AppImage, .deb, .rpm, and aarch64 variants. Always match what .github/workflows/cardano-node.yaml downloads.

WARNING: moog-head in /code/moog/tmp/ is a dev build. It reports the same --version as the release but encodes Duration differently ({"minutes":0} instead of 0), producing requests the oracle can't parse (UnknownUpdateValidationFailure). Never symlink moog-head as moog. Always download the release binary.

The workflow at .github/workflows/cardano-node.yaml downloads from cardano-foundation/moog releases — match that exactly.

Active Antithesis tenant: amaru-cardano (2026-05-26 →)

Antithesis migrated Cardano testing from the cardano tenant (cardano.antithesis.com / user cardano) to amaru-cardano (amaru-cardano.antithesis.com / user pragma). The moog-agent binary at 851aac3-dirty (= v0.4.1.2) hardcodes the old URL at src/User/Agent/PushTest.hs:246. The multi-tenant fix lives in https://github.com/cardano-foundation/moog/pull/93 (branch feat/multi-tenant-antithesis, head e6157f6); it reads MOOG_ANTITHESIS_LAUNCH_URL and MOOG_REGISTRY from env. Anything older than that PR cannot reach the new tenant; treat the deployed image: tag as part of the diagnosis.

Working credentials matrix (verify with curl before rotating anything — see feedback_split_config_test_outside.md):

Field	Value	Source
Antithesis URL	https://amaru-cardano.antithesis.com/api/v1/launch/amaru-cardano	env MOOG_ANTITHESIS_LAUNCH_URL
Antithesis user	pragma	env MOOG_ANTITHESIS_USER
Antithesis password	per secrets.yaml.antithesisPassword	secret file
GAR registry	us-central1-docker.pkg.dev/molten-verve-216720/cardano-repository (unchanged at the migration)	env MOOG_REGISTRY

Direct API check (outside moog) — succeeds with the right user/pw, 403 otherwise:

ssh agent 'PW=$(sudo awk "/^antithesisPassword:/{sub(/^antithesisPassword:[[:space:]]*\"?/,\"\");sub(/\"?[[:space:]]*$/,\"\");print}" /secrets/moog-agent/new/secrets.yaml) && \
  curl -sS -o /dev/null -w "%{http_code}\n" -u "pragma:$PW" \
    -X POST https://amaru-cardano.antithesis.com/api/v1/launch/amaru-cardano \
    -H "Content-Type: application/json" -d "{}"'
# 200 → creds OK; 403 → creds rejected (and same 403 with no -u, so 403 alone doesn't distinguish)

Identities

Role	Wallet	Set with
Requester	tmp/requester.json	being_requester
Agent	tmp/agent.json	being_agent
Oracle	tmp/oracle.json	being_oracle

Check current identity: being

Key commands

Command	Purpose
moog token --no-pretty	Full token state with pending requests
moog facts test-runs --whose cfhal	List test runs (run as being_requester to auto-decrypt the Antithesis report URL in the url field)
moog retract -w <wallet> -o <outref>	Retract a request you own
moog agent report-test -i <id> -w <wallet> --outcome unknown --duration 0 --url "reason"	Force-finish a stale test (manual override)
moog antithesis runs --no-pretty	List the Antithesis runs the agent reconciles against (see "How the agent reconciles results")
moog wallet info	Show wallet owner hash

Reading Antithesis runs directly (moog antithesis, v0.5.1.0+)

moog antithesis (added in v0.5.1.0) reaches the Antithesis report data through the Moog proxy — no browser, SSO cookie, or Playwright. All subcommands emit newline-delimited JSON; pair with --no-pretty | jq. Exit codes on runs: 0 success, 2 auth/SSO failure, 3 proxy/network failure, 4 invalid JSON from proxy.

Command	Purpose
moog antithesis runs [--limit N] [--cursor C]	List runs (paginated; --limit max 100). Each entry carries run_id (e.g. 46d4…-54-7), status (in_progress / completed / incomplete), and a JSON description with the testRun directory (e.g. testnets/cardano_node_governance).
moog antithesis run --run-id ID	Full details of one run. For incomplete runs it includes failure_moment {input_hash, vtime}; for completed runs it carries links.triage_report (the report URL).
moog antithesis properties --run-id ID	Property pass/fail for a completed run (404 while still in progress).
moog antithesis events --run-id ID [--q TEXT]	Stream matching events as NDJSON; free-text substring via --q; upstream caps at 50 results.
moog antithesis logs --run-id ID --input-hash H --vtime V	Logs at a specific moment. Needs a real moment — in_progress runs 400 (Invalid query parameters); a degenerate 0/0 failure moment returns empty.
moog antithesis build-logs --run-id ID	Stream the config-image build logs as NDJSON ({timestamp, text, stream}). Works regardless of run status — the fallback when there's no usable failure moment.

Find a run, then read it — e.g. the latest governance run's build logs:

# Locate the run id by directory (the banner from prod-setup.sh precedes the JSON,
# so select the JSON line explicitly):
moog antithesis runs --limit 100 --no-pretty | grep '^{"data"' \
  | jq -r '.data[] | select(.description|contains("governance"))
           | [.created_at, .status, .run_id] | @tsv' | head

# Moment logs need input_hash + vtime from `run`; build-logs need only the id:
moog antithesis build-logs --run-id 46d4033b0cbaec3715d6a7a77b4411cb-54-7 --no-pretty

Retract stuck requests (oracle crash-loop fix)

# 1. List pending requests on token
moog token --no-pretty | jq -c '.requests[] | {outref: .request.outputRefId, owner: .request.owner}'

# 2. Match owner to wallet (check with: moog wallet info)

# 3. Retract each owned request
being_requester  # or being_agent
moog retract -w $MOOG_WALLET_FILE -o "<outref>"

# 4. Oracle picks up changes automatically (no restart needed in theory,
#    but restart if it's stuck in a crash-loop reading stale state)

How the agent reconciles results (Antithesis API, v0.5.1.2+)

The agent derives outcomes from the Antithesis REST API, not email. The service loop (commit ca82717e, shipped in v0.5.1.2+) lists Antithesis runs and matches each to an on-chain test-run by its rendered description, then advances the on-chain phase. Reconciliation rules live in src/User/Agent/Antithesis/State.hs:

• pending on-chain → no matching Antithesis run yet ⇒ launch it; exactly one match ⇒ accept; many ⇒ flagged duplicate.

• accepted (running) on-chain → when the matched run reaches a terminal API status, write a finished fact with this mapping (terminalOutcome):

  Antithesis status	on-chain outcome
  completed	success — only if the run has a triage_report link; without it the agent keeps waiting (success is never attested without the report)
  incomplete	failure
  cancelled	failure
  starting / in_progress / unknown	keep waiting

• Terminal but no triage-report link (incomplete / cancelled): finished as failure anyway, with a deterministic synthetic URL antithesis://runs/<run_id>/no-triage-report (terminalNoReportOutcome, fix 04a74a40 / issue #138) so these no longer stay accepted forever.

So incomplete has no on-chain representation of its own — it is recorded as outcome: failure (the report URL if one exists, else the no-triage-report synthetic URL). The on-chain duration field is the requested hours, not actual runtime, so a truncated run is invisible once finished.

Clean up stale "running" tests

With the v0.5.1.2+ loop, terminal Antithesis runs auto-finish, so a genuinely stuck accepted fact now means either a correlation miss (the agent's description match failed — e.g. the run was launched by a different binary) or an agent running a pre-v0.5.1.2 binary. The manual escape hatch is unchanged:

# 1. Find accepted (running) tests on-chain
moog facts test-runs --whose cfhal --no-pretty | jq -c '.[] | select(.value.phase == "accepted") | {id: .id[:16], commit: .key.commitId[:8], try: .key.try}'

# 2. Confirm against the Antithesis API what status the run actually reached
moog antithesis runs --limit 100 --no-pretty | grep '^{"data"' | jq -r '.data[] | [.status, .run_id] | @tsv'

# 3. Report as unknown to finish them (manual override)
being_agent
moog agent report-test -i <full-test-run-id> -w $MOOG_WALLET_FILE --outcome unknown --duration 0 --url "stale-uncorrelated-run"

Infrastructure SSH

Host	Service	Container
oracle	moog-oracle v0.4.1.2	oracle-moog-oracle-1
agent	moog-agent (release binary; the v0.5.1.2+ loop reconciles via the Antithesis API)	agent-moog-agent-1

Compose lives at /opt/hal/infrastructure/moog/agent/docker-compose.yaml on the agent host. Edits go in there, not in /home/paolino/hal/... (which is a checkout, not the deployed copy).

Secrets layout

Agent host (agent)

/secrets/moog-agent/
├── old/                 # previous rotation generation (kept for rollback)
│   ├── secrets.yaml
│   └── docker/config.json
└── new/                 # currently mounted by compose
    ├── secrets.yaml     # githubPAT, antithesisPassword, trustedRequesters,
    │                    # mnemonics or encryptedMnemonics, walletPassphrase, slackWebhook,
    │                    # tokenId, mpfsHost, registry, antithesisUser, antithesisLaunchUrl,
    │                    # pollIntervalSeconds, minutes, wait
    └── docker/
        ├── config.json  # docker auth (auths block with base64 _json_key:<sa>)
        └── sa-key.json  # raw GCP service-account JSON (kept for regenerating config.json)

agent-wallet (a separate compose secret)
└── /secrets/moog-agent/new/agent.json (or wherever the agent-wallet.file points)

Compose: /opt/hal/infrastructure/moog/agent/docker-compose.yaml.

Oracle host (oracle)

/home/paolino/secrets/moog/oracle/
├── oracle.json          # oracle wallet (mnemonics OR encryptedMnemonics)
└── secrets.yaml         # githubPAT (cfhal-owned, distinct from agent's PAT),
                         # walletPassphrase (if oracle.json is encrypted)

Compose: /opt/hal/infrastructure/moog/oracle/docker-compose.yaml. Container env points MOOG_WALLET_FILE=/run/secrets/oracle-wallet and MOOG_SECRETS_FILE=/run/secrets/secrets at the in-container tmpfs paths the compose materialises from the two host files.

The oracle wallet CANNOT be rotated on a live token. The on-chain token's state.owner is permanently bound to the wallet's pubkey-hash at mint time (moog token state --no-pretty | jq .state.owner). Rotating the oracle wallet means burning the token and reminting — which loses the entire test-run history. If the oracle mnemonics leak (as happened 2026-05-28 — see lessons), the only mitigations are (a) moog wallet encrypt so on-disk material isn't immediately usable without the passphrase, and (b) accepting that the preprod blast radius is limited to fraudulent inclusions on this token's history and drainable preprod ADA — no mainnet impact. Treat the oracle wallet file as if it were a mainnet hot key: never cat, never head, never scp it off the host.

old/ and new/ may have different githubPAT / antithesisPassword / docker config.json values. When something fails on auth, diff the two — a half-applied rotation is a real failure mode.

docker/config.json is NOT the raw SA key. It must be a real docker config:

{
  "auths": {
    "us-central1-docker.pkg.dev": {