test-e2e-podman

name	test-e2e-podman
description	Run end-to-end tests for the podman backend using real container and API calls

End-to-End Podman Backend Test

Run full lifecycle tests of the podman backend across harnesses and auth modes. All tests run inside a CI container with nested podman.

Each section below is independent. Run whichever sections match your environment and skip the rest.

Before you start

Present the following default container images to the user and ask them to confirm or override each one before proceeding:

Role	Default image	Variable
CI image	quay.io/aipcc/agentic-ci/podman	$CI_IMAGE
Claude runner	quay.io/aipcc/agentic-ci/claude-runner:latest	$CLAUDE_IMAGE
OpenCode runner	quay.io/aipcc/agentic-ci/opencode-runner:latest	$OPENCODE_IMAGE

Also ask for:

• API key file — path to a local file containing an Anthropic API key (one line, no whitespace). Needed for Sections B and D. Store as $API_KEY_FILE.

Per-section requirements:

• Vertex AI auth (Sections A, E): GCP ADC credentials at ~/.config/gcloud/application_default_credentials.json and ANTHROPIC_VERTEX_PROJECT_ID + CLOUD_ML_REGION set in the environment.
• API key auth (Sections B, D, F): The API key file above.

Container setup

All agentic-ci commands run inside the CI container. The container needs --privileged for nested podman (the podman backend starts a second container inside the CI container).

Start the test container

podman rm -f podman-e2e 2>/dev/null || true
podman run -d --name podman-e2e \
  --privileged \
  -v "$(pwd):/workspace:ro,z" \
  -v ~/.config/gcloud:/host-gcloud:ro,z \
  -v "$API_KEY_FILE:/host-api-key:ro,z" \
  $CI_IMAGE \
  sleep infinity

Prepare the environment

podman exec podman-e2e bash -c '
  mkdir -p ~/.config/gcloud
  cp /host-gcloud/application_default_credentials.json ~/.config/gcloud/
  cd /workspace && uv pip install --system --no-cache .
'

Verify:

• podman exec podman-e2e agentic-ci --help prints usage
• podman exec podman-e2e podman --version shows podman is available

Cleanup between tests

Reset the inner agentic-ci container between sections:

podman exec podman-e2e bash -c 'podman rm -f agentic-ci 2>/dev/null || true'

Models

Use haiku to keep cost down. The model name format varies by harness:

Harness	Vertex AI model	API key model
Claude Code	claude-haiku-4-5	claude-haiku-4-5
OpenCode	(not supported via Vertex)	anthropic/claude-haiku-4-5-20251001

---

Section A: Claude Code + Vertex AI

Requires GCP ADC credentials and ANTHROPIC_VERTEX_PROJECT_ID.

A1. Setup

podman exec \
  -e ANTHROPIC_VERTEX_PROJECT_ID=<your-project-id> \
  -e CLOUD_ML_REGION=global \
  -e CLAUDE_IMAGE="$CLAUDE_IMAGE" \
  podman-e2e bash -c '
    agentic-ci setup --image "$CLAUDE_IMAGE"
  '

Verify:

• Output shows Auth: Vertex AI
• Output says --- Podman container started ---
• podman exec podman-e2e podman ps --filter name=agentic-ci shows the inner container running

A2. Run (streaming, no OTEL)

podman exec \
  -e ANTHROPIC_VERTEX_PROJECT_ID=<your-project-id> \
  -e CLOUD_ML_REGION=global \
  -e CLAUDE_IMAGE="$CLAUDE_IMAGE" \
  podman-e2e bash -c '
    agentic-ci run "Respond with exactly: A2_OK" \
      --image "$CLAUDE_IMAGE" \
      --model claude-haiku-4-5 --no-otel
  '

Verify:

• Output says --- Podman container already running --- (reuses container from setup)
• Streaming output shows colored text blocks (thinking, tool calls, Claude response)
• Claude's response contains A2_OK
• Exit code is 0

A3. Run (streaming, with OTEL)

podman exec \
  -e ANTHROPIC_VERTEX_PROJECT_ID=<your-project-id> \
  -e CLOUD_ML_REGION=global \
  -e CLAUDE_IMAGE="$CLAUDE_IMAGE" \
  podman-e2e bash -c '
    agentic-ci run "Respond with exactly: A3_OK" \
      --image "$CLAUDE_IMAGE" \
      --model claude-haiku-4-5
  '

Verify:

• Container is still reused (already running)
• OTEL collector starts on a dynamic port
• Streaming output works
• Claude's response contains A3_OK
• OTEL Token/Cost Summary prints at the end with token counts and USD costs
• Exit code is 0

A3a. Verify trace records for mlflow-push

Check that the OTEL JSONL log contains /v1/traces records — these are what agentic-ci mlflow-push needs:

podman exec podman-e2e bash -c '
  JSONL=$(ls -t /tmp/agentic-ci-run.*/claude-otel.jsonl 2>/dev/null | head -1)
  echo "JSONL file: $JSONL"
  grep -c "/v1/traces" "$JSONL"
'

Verify:

• The JSONL file exists
• The trace count is at least 1
• Optionally inspect a trace record:

  podman exec podman-e2e bash -c '
    JSONL=$(ls -t /tmp/agentic-ci-run.*/claude-otel.jsonl 2>/dev/null | head -1)
    grep "/v1/traces" "$JSONL" | head -1 | python3 -m json.tool
  '
  

  • Record has "path" containing /v1/traces
  • Record has "payload" with "resourceSpans" array

A4. Run (no streaming)

podman exec \
  -e ANTHROPIC_VERTEX_PROJECT_ID=<your-project-id> \
  -e CLOUD_ML_REGION=global \
  -e CLAUDE_IMAGE="$CLAUDE_IMAGE" \
  podman-e2e bash -c '
    agentic-ci run "Respond with exactly: A4_OK" \
      --image "$CLAUDE_IMAGE" \
      --model claude-haiku-4-5 --no-streaming --no-otel
  '

Verify:

• No formatted streaming output (no colored blocks, no tool summaries)
• Exit code is 0

A5. Stop

podman exec \
  -e CLAUDE_IMAGE="$CLAUDE_IMAGE" \
  podman-e2e bash -c '
    agentic-ci stop --image "$CLAUDE_IMAGE"
  '

Verify:

• Output says --- Podman container stopped ---
• podman exec podman-e2e podman ps -a --filter name=agentic-ci shows no container

---

Section B: Claude Code + API Key

Run cleanup first.

B1. Setup and run

podman exec \
  -e "ANTHROPIC_API_KEY=$(cat "$API_KEY_FILE")" \
  -e CLAUDE_IMAGE="$CLAUDE_IMAGE" \
  podman-e2e bash -c '
    agentic-ci run "Respond with exactly: B1_OK" \
      --image "$CLAUDE_IMAGE" \
      --model claude-haiku-4-5 --no-otel
  '

Verify:

• Output shows Auth: API key (not Vertex AI)
• Streaming output works
• Claude's response contains B1_OK
• Exit code is 0

B2. Run (streaming, with OTEL)

podman exec \
  -e "ANTHROPIC_API_KEY=$(cat "$API_KEY_FILE")" \
  -e CLAUDE_IMAGE="$CLAUDE_IMAGE" \
  podman-e2e bash -c '
    agentic-ci run "Respond with exactly: B2_OK" \
      --image "$CLAUDE_IMAGE" \
      --model claude-haiku-4-5
  '

Verify:

• OTEL collector starts and Token/Cost Summary prints at the end
• Claude's response contains B2_OK
• Exit code is 0

B2a. Verify trace records for mlflow-push

Same check as A3a — verify the JSONL log has /v1/traces records:

podman exec podman-e2e bash -c '
  JSONL=$(ls -t /tmp/agentic-ci-run.*/claude-otel.jsonl 2>/dev/null | head -1)
  echo "JSONL file: $JSONL"
  grep -c "/v1/traces" "$JSONL"
'

Verify:

• The JSONL file exists
• The trace count is at least 1

B3. Stop

podman exec \
  -e CLAUDE_IMAGE="$CLAUDE_IMAGE" \
  podman-e2e bash -c '
    agentic-ci stop --image "$CLAUDE_IMAGE"
  '

Verify:

• Output says --- Podman container stopped ---

---

Section C: OpenCode + Vertex AI

Requires GCP ADC credentials.

Run cleanup first.

C1. Run

podman exec \
  -e ANTHROPIC_VERTEX_PROJECT_ID=<your-project-id> \
  -e CLOUD_ML_REGION=global \
  -e OPENCODE_IMAGE="$OPENCODE_IMAGE" \
  podman-e2e bash -c '
    agentic-ci run "Respond with exactly: C1_OK" \
      --harness opencode \
      --image "$OPENCODE_IMAGE" \
      --model google-vertex/claude-haiku-4-5@20251001 \
      --no-otel
  '

Verify:

• Output shows Harness: OpenCode and Auth: Vertex AI
• Streaming output shows agent activity
• Exit code is 0

C2. Stop

podman exec \
  -e OPENCODE_IMAGE="$OPENCODE_IMAGE" \
  podman-e2e bash -c '
    agentic-ci stop --harness opencode --image "$OPENCODE_IMAGE"
  '

Verify:

• Output says --- Podman container stopped ---

---

Section D: OpenCode + API Key

Requires ANTHROPIC_API_KEY (via API key file).

Run cleanup first.

D1. Run

podman exec \
  -e "ANTHROPIC_API_KEY=$(cat "$API_KEY_FILE")" \
  -e OPENCODE_IMAGE="$OPENCODE_IMAGE" \
  podman-e2e bash -c '
    agentic-ci run "Respond with exactly: D1_OK" \
      --harness opencode \
      --image "$OPENCODE_IMAGE" \
      --model anthropic/claude-haiku-4-5-20251001 \
      --no-otel
  '

Verify:

• Output shows Harness: OpenCode and Auth: API key
• Streaming output shows agent activity
• Exit code is 0

D2. Stop

podman exec \
  -e OPENCODE_IMAGE="$OPENCODE_IMAGE" \
  podman-e2e bash -c '
    agentic-ci stop --harness opencode --image "$OPENCODE_IMAGE"
  '

Verify:

• Output says --- Podman container stopped ---

---

Section E: Skill execution — Vertex AI

Verifies that a plugin skill loads and executes correctly inside the runner container. Uses the git-shallow-clone skill from the odh-ai-helpers plugin to clone a repo into /tmp and confirm the result.

Requires GCP ADC credentials and ANTHROPIC_VERTEX_PROJECT_ID.

Run cleanup first.

E1. Run

podman exec \
  -e ANTHROPIC_VERTEX_PROJECT_ID=<your-project-id> \
  -e CLOUD_ML_REGION=global \
  -e CLAUDE_IMAGE="$CLAUDE_IMAGE" \
  podman-e2e bash -c '
    agentic-ci run \
      --image "$CLAUDE_IMAGE" \
      --model claude-haiku-4-5 --no-otel \
      "Use the /odh-ai-helpers:git-shallow-clone skill to shallow-clone https://github.com/opendatahub-io/agentic-ci.git into /tmp/agentic-ci. After the clone completes, run: ls /tmp/agentic-ci and print the output, then respond with exactly: SKILL_OK"
  '

Verify:

• Plugins: line includes odh-ai-helpers
• Output shows Skill tool invocation for odh-ai-helpers:git-shallow-clone
• Output shows a Bash tool call running git clone
• Output shows ls /tmp/agentic-ci with repo contents (e.g. src, pyproject.toml, Makefile)
• Agent response contains SKILL_OK
• Agent exit code: 0

E2. Stop

podman exec \
  -e CLAUDE_IMAGE="$CLAUDE_IMAGE" \
  podman-e2e bash -c '
    agentic-ci stop --image "$CLAUDE_IMAGE"
  '

---

Section F: Skill execution — API Key

Same test as Section E but with API key auth.

Run cleanup first.

F1. Run

podman exec \
  -e "ANTHROPIC_API_KEY=$(cat "$API_KEY_FILE")" \
  -e CLAUDE_IMAGE="$CLAUDE_IMAGE" \
  podman-e2e bash -c '
    agentic-ci run \
      --image "$CLAUDE_IMAGE" \
      --model claude-haiku-4-5 --no-otel \
      "Use the /odh-ai-helpers:git-shallow-clone skill to shallow-clone https://github.com/opendatahub-io/agentic-ci.git into /tmp/agentic-ci. After the clone completes, run: ls /tmp/agentic-ci and print the output, then respond with exactly: SKILL_OK"
  '

Verify:

• Output shows Auth: API key
• Plugins: line includes odh-ai-helpers
• Output shows Skill tool invocation for odh-ai-helpers:git-shallow-clone
• Output shows ls /tmp/agentic-ci with repo contents (e.g. src, pyproject.toml, Makefile)
• Agent response contains SKILL_OK
• Agent exit code: 0

F2. Stop

podman exec \
  -e CLAUDE_IMAGE="$CLAUDE_IMAGE" \
  podman-e2e bash -c '
    agentic-ci stop --image "$CLAUDE_IMAGE"
  '

---

Final cleanup

podman rm -f podman-e2e

Running the full suite

Execute sections in order (A through F), running the cleanup step before each section. Skip sections whose prerequisites are not met. If any step fails, stop and investigate. Check inner container logs with:

podman exec podman-e2e podman logs agentic-ci