copilot-sdk-e2e-dev

name	copilot-sdk-e2e-dev
description	Spin up a live local agent-meow server and exercise the GitHub Copilot SDK harness end-to-end — build copilot agents, run real turns, smoke-test, and bug-bash. Load when developing, testing, or debugging the copilot harness (agent_meow/inner/copilot_executor.py, copilot_harness.py, agent_meow/onboarding/copilot_auth.py) or its auth / model / tool-bridge behavior.

Copilot SDK harness: end-to-end dev & testing

The copilot harness drives the GitHub Copilot SDK (github-copilot-sdk, imported as copilot) — a persistent CopilotClient + CopilotSession per agent-meow conversation — and bridges agent-meow's sys_* tools into Copilot as SDK Tools. The Python SDK bundles the Copilot CLI binary it drives as a backing server, so there is no separate @github/copilot install. This skill is the proven recipe for running it for real against a live local server — not just the unit tests.

The harness runs as a local runner from your current checkout, so omni run <bundle> --server <url> exercises exactly the code you're on.

Prerequisites (check these first)

1. You're on the branch you want to test. The copilot harness is an optional extra — install it (without disturbing other extras) with uv sync --frozen --extra dev --extra copilot. NB: a bare uv run --frozen --extra dev re-syncs the venv and prunes the copilot SDK; for live testing call .venv/bin/omni / .venv/bin/python directly and avoid uv run mid-session.
2. The SDK is installed: .venv/bin/python -c "import copilot; print(copilot.__file__)".
3. A GitHub token with Copilot access is configured. Copilot needs a fine-grained PAT with the "Copilot Requests" permission, or an OAuth token from the GitHub CLI / Copilot CLI app (classic ghp_ PATs are rejected). Verify (booleans only — never print the token):

   .venv/bin/python -c "from agent_meow.onboarding.copilot_auth import copilot_github_token_configured; import os; print('config:', copilot_github_token_configured(), 'env:', bool(os.environ.get('GH_TOKEN') or os.environ.get('COPILOT_GITHUB_TOKEN')))"
   

   If both are False, run omni setup and register a Copilot token, or export GH_TOKEN=$(gh auth token) (when gh is logged into an account with Copilot). Check the account's entitlement with gh api /copilot_internal/user (look for chat_enabled/cli_enabled).
4. Network egress to GitHub's Copilot backend. A turn that hangs or fails to connect on a locked-down host is usually egress, not a harness bug.

Step 1 — start a local server

cd /path/to/agent-meow
.venv/bin/omni server --port 7788 --no-open    # foreground; or `omni server start` for detached
curl -s http://127.0.0.1:7788/health           # {"status":"ok"}

Use the URL below as $SERVER.

Step 2 — build a copilot agent bundle

A spec with spec_version must be a directory containing config.yaml — not a single .yaml file. Minimal copilot agent:

mkdir -p /tmp/copilot-dev
cat > /tmp/copilot-dev/config.yaml <<'YAML'
spec_version: 1
name: copilot-dev
description: Copilot SDK dev/test agent.
executor:
  type: agent-meow
  config:
    harness: copilot
    # model: gpt-5-mini      # optional; omit for Copilot auto-select
prompt: |
  You are a terse test agent. Answer in as few words as possible.
YAML

For sub-agents, tools, guardrails/policies, copy the field shapes from examples/polly/config.yaml and examples/debby/config.yaml. (Declare policies under guardrails.policies: — a top-level policies: key is silently dropped on the spec_version + config.yaml path.)

Step 3 — run a turn (and smoke-test)

SERVER=http://127.0.0.1:7788
timeout 280 .venv/bin/omni run /tmp/copilot-dev \
  -p "Reply with exactly the single word: PONG" \
  --server "$SERVER" 2>&1

A healthy run prints connection lines then the reply (PONG). If that works, the full stack is good: token, egress, bundled CLI, harness.

• Shell / file tools: add --tools coding.
• Specific model: add --model gpt-5-mini (or claude-haiku-4.5, auto).

Targeted scenarios

Goal	How
Native tools (shell/edit/read)	--tools coding, prompt to create→read→edit a file; confirm it actually touches disk
Bridged sys_* / sub-agent dispatch	declare a sub-agent (harness copilot so auth is satisfied), prompt the parent to delegate — exercises the SDK Tool async-handler bridge into _tool_executor
Model routing	run the same bundle with several --model values; an unknown id fails loud, a databricks-* id is dropped to auto with a warning
LLM-phase policy	add a guardrail that denies a keyword; confirm PHASE_LLM_REQUEST/PHASE_LLM_RESPONSE blocks it
Concurrency / leaks	fire several omni run … & at once; then pgrep -af "copilot/bin/copilot" to check for orphaned bundled-CLI subprocesses

Running polly (or any orchestrator) on a copilot brain

The copilot harness can serve as an async orchestrator brain (polly / debby), not just a standalone agent — it dispatches to sub-agents via the bridged sys_* tools and synthesizes their results. Two ways to exercise it:

1. Committed regression guard (brain smoke). tests/e2e/test_polly_copilot_e2e.py boots a local server from your checkout and runs examples/polly with --harness copilot --model auto, asserting the brain boots and replies. It is skipped unless a Copilot token is configured (so CI without one skips it). Run it with:

.venv/bin/python -m pytest -o addopts="" tests/e2e/test_polly_copilot_e2e.py -v

2. Full orchestration (dispatch → collect → synthesize). Use the polly-e2e-dev driver (in the internal agent-framework clone) — it boots a local server, polls the AP API, auto-answers elicitations, and asserts the fan-out. Drive the brain on copilot with --brain-harness copilot, and always pass a Copilot-catalog --brain-model (auto, claude-haiku-4.5, gpt-5-mini): the driver's default --brain-model is a Claude id that Copilot (no Databricks gateway) can't route. From the agent-framework clone:

.venv/bin/python .claude/skills/polly-e2e-dev/polly_driver.py \
  --local --code-dir <this-worktree> \
  --cuj smoke --brain-harness copilot --brain-model auto      # brain only
# --cuj fanout  …  and  --cuj review-pr --repo JZKK720/agent-meow --pr <n>  …
#   exercise real sub-agent dispatch (claude_code + codex) under a copilot brain.

All three CUJs (smoke / fanout / review-pr) pass on a copilot brain (verified live: fanout dispatched 8 sub-agents, 8/8 OK + a synthesis). Note omni run -p exits after the dispatch turn (the brain parks until woken), so a sub-agent's final answer lands server-side — read it over the AP API (GET /v1/sessions/{id}/items, child sessions), not just stdout.

Gotchas (these cost real time)

1. config.yaml's server: defaults to a remote server. Omitting --server sends your turn to that remote deploy — which may be stale and reject the copilot harness with executor.config.harness: must be one of […]. Always pass --server http://127.0.0.1:<port>. (If a local server rejects copilot, it's running stale code — restart it from your checkout.)
2. A spec with spec_version must be a directory + config.yaml, never a single .yaml file.
3. Copilot needs a GitHub token (fine-grained PAT w/ Copilot Requests, or a gh/Copilot-CLI OAuth token). Resolution precedence: spec executor.auth (api_key) > stored copilot: config block (omni setup) > ambient COPILOT_GITHUB_TOKEN / GH_TOKEN / GITHUB_TOKEN. Classic ghp_ rejected.
4. No Databricks gateway. Copilot talks only to GitHub's backend, so a databricks-* model is silently resolved to Copilot's auto-select — it will not route through the AI Gateway like claude-sdk/codex/pi.
5. Use a model id from the account's catalog. free_limited offers auto, claude-haiku-4.5, gpt-5-mini. Run .venv/bin/python + client.list_models() to discover the live set; an unknown id fails loud (server-side failed session).
6. Turns take 30–90s — always wrap in timeout 280.
7. Never print/echo the GitHub token in logs or commands.

Code & tests

• Executor (SDK bridge): agent_meow/inner/copilot_executor.py
• Wrap (HARNESS_COPILOT_ env → executor):* agent_meow/inner/copilot_harness.py
• Auth / token resolution: agent_meow/onboarding/copilot_auth.py
• Spawn env: _build_copilot_spawn_env in agent_meow/runtime/workflow.py

uv run --frozen --extra dev python -m pytest \
  tests/inner/test_copilot_executor.py \
  tests/inner/test_copilot_harness.py \
  tests/runtime/test_copilot_spawn_env.py \
  tests/onboarding/test_copilot_auth.py -q

Bug-bash (fan out)

To stress the harness, run several scenario probes in parallel — each builds a bundle and runs real turns against the same $SERVER, then reports what broke. Highest-value targets: the Tool async-handler bridge (hangs / lost tool results / errors reported as success), model routing, policy enforcement, streamed-output rendering, and orphaned bundled-CLI processes after teardown. Cross-check the AP API (GET /v1/sessions/{id}/items) — a start failure can exit 0 with empty stdout while the server records a failed session.

Known sharp edges (found via live bug-bash — "as of this writing")

• Native tools bypass on:[tool_call] policies and aren't recorded. Copilot's built-in create/view/edit/bash run inside the SDK, so an on:[tool_call] DENY guardrail (e.g. blast_radius) never sees them, and they leave no function_call item in the transcript (only streamed narration). Bridged sys_* tools ARE gated and recorded. Gate Copilot's built-ins at the LLM phase (PHASE_LLM_REQUEST/RESPONSE, which fire) or via the OS-env sandbox — not on:[tool_call]. (Same shape as the cursor harness.)
• Copilot fails loud (unlike cursor's swallowed start failures). Bad token, empty/invalid model, and unknown model ids all exit non-zero with a clear error AND a server-side failed session + error item — verified, not swallowed.
• omni run -p against an async orchestrator exits after the dispatch turn, so a delegated sub-agent's final answer is persisted server-side but may not reach stdout in one-shot mode. Read the session over the AP API to see it.
• Non-graceful exit can orphan the bundled CLI. Graceful teardown reaps it (client.stop()); after a SIGKILL/hard-exit, sweep pgrep -af "copilot/bin/copilot".

Cleanup

.venv/bin/omni server stop        # or kill the foreground `omni server`
rm -rf /tmp/copilot-dev           # remove scratch bundles
pgrep -af "copilot/bin/copilot"   # confirm no orphaned bundled-CLI subprocesses linger