threadlight-local-test

# Threadlight — Local Test Loop (no azd up) ## Presenter-ready integration contract For the explicit [presenter-ready profile](../../docs/presenter-ready.md), exercise the **exact packaged SDK/framework, host/client and process adapter** selected in the process handoff. Cover response shapes (data-plane, not management-plane substitutes), legitimate metadata reads, real retry/dispatch, pending lifecycle, terminal stream errors, persistence, independent readback, reopening and recovery. Bind outputs to the dependency lock/source and record the layer actually run: offline, native, hosted and quality/human evidence are different. Pattern 0 remains a quick iteration option, not a universal runtime or durable result store. Its in-memory data resets on recreation and cannot satisfy promised save/reopen. Provisional stream text is not success before terminal validation; a late failure replaces successful-looking result text. Reuse valid focused evidence; no blanket SDK upgrade, broad rerun or business replay for screenshots. Run a generated PoC entirely on your dev box so you can iterate on **tools**, **prompts**, and **workspace UI** in seconds — not in the 20-30 min round-trip of `azd deploy`. Designed for use **inside** GitHub Copilot CLI, Cowork, or Clawpilot, where you want to hand the running agent / MCP server to the CLI itself for hands-on testing. > **What this skill is NOT.** This is not a "make Foundry run on > your laptop" skill — Foundry hosted-agent runtime stays in Azure. > What this skill *is* is the recipe for running the **same agent > code, the same MCP server code, the same workspace HTML** on > localhost so you can debug fast, then redeploy via > `threadlight-deploy` when you're happy. --- ## When to use which pattern | Pattern | What it runs | When to use | |---------|--------------|-------------| | **0. Quickstart** (default) | `python -m threadlight_quickstart` → MAF `Agent + SkillsProvider` + JSON stub tools + Streamlit UI on `localhost:8501` | **First reach-for after `threadlight-design`.** Closes the design → screen-shareable demo loop to <30 min. Zero Docker, zero MCP server boot, one LLM dep (Foundry project OR AOAI deployment OR GitHub Models via `GITHUB_TOKEN`). | | **1. MCP-direct** (CLI ↔ MCP) | Just the PoC's FastMCP server on `localhost:8000`; CLI calls tools natively | You're iterating on **MCP tool implementation** (DB queries, business rules, error handling). The CLI itself is the agent. | | **2. Smoke-client** (CLI → Python → Agent) | The PoC's `Agent + FoundryChatClient` invoked via `agent.run_async()` from a smoke script | You're iterating on the **prompt** or the **agent's tool-orchestration** behaviour. Skips the `ResponsesHostServer` HTTP layer. | | **3. Local-stack** (compose) ⚠️ | All of: MCP server + Cosmos emulator + workspace UI on nginx + (optional) Search mock | End-to-end smoke before redeploying. **Linux / Windows x86 only** — Cosmos emulator container is fragile on macOS ARM; use Pattern 0 there. | Reach for **Pattern 0** first. Drop to **Pattern 1** when you need to iterate on the real MCP server. Use **Pattern 2** for headless prompt tuning that needs to be driven from the CLI. **Pattern 3** is a pre-deploy parity check on platforms where the Cosmos emulator container actually works. --- ## Pattern 0 — Quickstart (default) The new default for "I just finished `threadlight-design`, now I want to see it run." Consumes what design already emits — `specs/sample-data/`, `src/agent/skills/<name>/SKILL.md` — and gives you a MAF agent + a chat UI on `localhost:8501` without ever touching Docker, the MCP server, or a real Cosmos. > **Kratos-export mode.** Pattern 0 also boots a **Kratos-exported project** > (`src/hosted-agent/` + `use-cases/<x>/` — see > [`docs/KRATOS-BRIDGE.md`](../../docs/KRATOS-BRIDGE.md)) — the hosted-agent > contract is identical, so the same MAF `Agent + SkillsProvider` recipe applies. > Two path differences: load skills from the resolved skills root > `use-cases/<x>/skills/` (auto-detected; `--skills-root` to override) instead of > `src/agent/skills/`, and seed the in-memory store from the export's `mocks/` > directory in place of `specs/sample-data/`. The agent identity/system prompt > comes from `use-cases/<x>/SYSTEM_PROMPT.md`. No `azd up`, same fast loop. ### What you need | Need | Why | One-time? | |------|-----|-----------| | **Python ≥ 3.10** + `pip` (or `uv`) | Run the reference package | Yes | | **A threadlight-designed PoC** (the cwd at minimum has `specs/sample-data/*.json`) | Pattern 0 auto-discovers it | Per PoC | | **One LLM endpoint** — Foundry project URL **OR** Azure OpenAI deployment **OR** GitHub Models (zero Azure) | The only external dep | Per tenant (shared GBB sandbox is fine; or just `gh auth token` for GitHub Models) | | **Auth** — `az login` to the LLM tenant (Foundry/AOAI) **OR** `GITHUB_TOKEN` (GitHub Models) | Credential for the LLM call | Per tenant | > **No Docker. No Cosmos emulator. No MCP server boot.** Pattern 0 > replaces all three with an in-memory dict-of-records loaded from > `specs/sample-data/<entity>.json` and three CRUD tools per entity > (`list_<entity>`, `get_<entity>`, `update_<entity>`). ### The bootstrap (one-time per PoC) ```bash # 1) Install the quickstart package once (editable, from the catalog) pip install -e <awesome-gbb>/skills/threadlight-local-test/references/quickstart # 2) Drop the env template into the PoC and edit it cp <awesome-gbb>/skills/threadlight-local-test/references/quickstart/.env.local.example .env.local $EDITOR .env.local # set LLM_BACKEND: foundry (default), aoai, or copilot (GitHub Models, no Azure) echo .env.local >> .gitignore # 3) Sanity-check the wiring without a live LLM round-trip (<5s) python -m threadlight_quickstart --check ``` ### The loop (every time) ```bash az login --tenant <dev-tid> # per azure-tenant-isolation skill python -m threadlight_quickstart # Streamlit on http://localhost:8501 ``` Or with the demo-prompt pump pre-loaded so you can step through the prep-guide acts hands-free: ```bash python -m threadlight_quickstart --simulator ``` `--simulator` reads prompts in priority order from: 1. `<poc-root>/tests/demo-prompts.txt` (one prompt per line; `#` comments allowed) 2. `<poc-root>/specs/prep-guide.html` § *Demo Script* (regex on `<strong>Type this:</strong>` blocks) > **`.env.local` is auto-loaded.** The CLI parses `<poc-root>/.env.local` > on every launch and injects the keys into the process env (only ones > not already set in the shell). No need to `source` it manually. > Disable with `THREADLIGHT_QUICKSTART_NO_TRANSCRIPT=1` if you don't > want the side-effect. > **Every UI turn appends to `<poc-root>/tests/quickstart.jsonl`.** > Shape matches what [`foundry-evals`](https://github.com/aiappsgbb/awesome-gbb/blob/main/skills/foundry-evals/SKILL.md) > consumes — `{ts, query, response}` per row. Run a few Pattern 0 > demos, then promote the JSONL into your Foundry eval dataset > without reshaping. Disable with `THREADLIGHT_QUICKSTART_NO_TRANSCRIPT=1`. > Add `tests/quickstart.jsonl` to your PoC's `.gitignore` (it's local > demo state, not a fixture). ### How the tools come from your SPEC For every `specs/sample-data/<entity>.json` discovered, Pattern 0 registers three MAF `@tool`-decorated callables backed by an in-memory `InMemoryStore`: | Tool | Returns | Notes | |------|---------|-------| | `list_<entity>(filters=None)` | `list[dict]` | Equality match on each field in the filters object; no filter → all records | | `get_<entity>(id)` | `dict \| None` | Lookup by record id | | `update_<entity>(id, fields)` | `dict` | Explicit fields object; mutates the in-memory snapshot; **reset every launch** | The named object parameters are intentional: the actual native SDK schema omits arbitrary `**kwargs`, which otherwise silently drops filters and update fields. Existing Python store methods retain their keyword API; native tool callers use the explicit `filters` / `fields` objects. The agent's `SkillsProvider` discovers `src/agent/skills/<name>/SKILL.md` under `from_paths(skills_dir)` and loads bodies on demand. Follow the [autonomous MAF skill-read contract](../_shared/maf-skill-approval.md): only trusted local instructions/resources bypass the provider's read approval; scripts and other tools keep their policies. Unresolved native approvals raise `approval_required`, surfaced by the existing UI error path rather than an empty successful turn. This local configuration is not proof of hosted parity. ### Custom tools (when CRUD isn't enough) For cross-entity joins, derived fields, business rules — drop a `tests/quickstart_tools.py` next to the PoC, exposing: ```python def register(tools: list, stores: dict) -> list: """Return extra MAF tools to append to the auto-generated CRUD set.""" from agent_framework import tool @tool def reassign_urgent(assignee: str) -> int: urgent = stores["tickets"].list_all(severity="urgent") for row in urgent: stores["tickets"].update(row["id"], assignee=assignee) return len(urgent) return [reassign_urgent] ``` `agent_wiring` auto-discovers and calls `register(tools, stores)` after the CRUD triple is built. ### What ships in the reference package ``` references/quickstart/ ├── pyproject.toml # pip-installable; pins streamlit + agent-framework ├── threadlight_quickstart/ │ ├── __main__.py # `python -m threadlight_quickstart` │ ├── cli.py # argparse: --check, --info, --simulator, --port │ ├── discover.py # walks up cwd for the canonical PoC layout │ ├── agent_wiring.py # Agent + SkillsProvider + tool registration │ ├── stub_tools.py # InMemoryStore + CRUD tool factory │ ├── simulator.py # demo-prompt cursor │ └── ui_streamlit.py # the Streamlit chat page ├── .env.local.example # env-var template ├── Makefile.demo # drop-in for PoCs that want `make demo` ├── fixture-poc/ # 1-skill, 1-entity toy PoC (smoke target) └── tests/ # pytest against fixture-poc ``` ### When Pattern 0 is **not** the right answer - **You need the real React workspace UI render** → Pattern 1 + run the PoC's own `npm run dev:workspace` separately. Pattern 0 ships Streamlit only on purpose (zero Node toolchain). - **You're debugging the real MCP server** → Pattern 1. Pattern 0 bypasses the MCP layer entirely. - **You need real Cosmos / Search semantics** (ranking, partition keys, RU shape) → Pattern 3 on Linux/Windows x86, or `azd up` to a dev sub. - **You need multi-agent orchestration** → Pattern 2. --- ## Prerequisites (one-time, dev box) | Need | Why | Install | |------|-----|---------| | **Python 3.13** + `uv` | Run agent/MCP code | [uv install](https://docs.astral.sh/uv/) | | **Docker Desktop** (or Rancher) | Cosmos emulator + nginx | Standard install | | **Azure OpenAI deployment** of `gpt-5.4-mini` (or any model), **OR** GitHub Models via `GITHUB_TOKEN` | Agent needs a real LLM | Any AOAI account, or just a GitHub account for GitHub Models. **The skill does NOT require Foundry locally.** | | **`az` logged in** to the AOAI tenant | DefaultAzureCredential in the agent code resolves to your `az` token | `az login --tenant <tid>` (per `azure-tenant-isolation`) | | **GitHub Copilot CLI** ≥ 1.0.40 | For Pattern 1 (MCP-direct) | `gh extension install github/gh-copilot-cli` | > **Bring-Your-Own-Foundry option.** If you have a Foundry project, > Patterns 2 and 3 can use `FoundryChatClient(project_endpoint=...)` > exactly as in production. If you don't, swap to plain `OpenAIClient` > pointed at AOAI directly — the agent code is identical aside from > the client constructor. See `references/local-stack/local_smoke.py` > for both forms. --- ## Pattern 1 — MCP-direct (Copilot CLI ↔ local MCP) The cleanest dev loop for **tool development**. The CLI itself acts as the agent; you call tools natively from natural language. ### Setup (3 lines) ```powershell # 1. Run the PoC's MCP server locally cd <poc-root>/src/mcp_server uv run python main.py # binds to http://localhost:8000/mcp # 2. Register it with Copilot CLI (per-user; persists) copilot mcp add <poc-name>-local --url http://localhost:8000/mcp # 3. Restart the CLI session copilot ``` ### Iteration loop In the CLI, ask: `"call list_open_disputes; what do you see?"` → the CLI invokes the local tool directly, you see the JSON response, you tweak `mcp_server/main.py`, the dev-loop reload picks it up (FastMCP supports `--reload`), you re-ask the CLI. **Why this is fast:** zero LLM round-trips for the dev parts you don't care about (no agent prompt to debug here); the CLI's built-in agent calls your tool once, you read the JSON, you fix. > **Pitfall.** If your MCP server reads from Azure resources > (Cosmos, Search) using `DefaultAzureCredential`, the local > process needs to authenticate to those — either use the dev > stack from Pattern 3 (local Cosmos), or `az login` against the > tenant that owns the real cloud resources. See > `references/cli-integration/copilot_mcp_register.md` for the > full setup including a localhost MCP that reads from cloud > Cosmos via your `az` token. See `references/cli-integration/copilot_mcp_register.md`. --- ## Pattern 2 — Smoke-client (direct `agent.run_async()`) For **prompt tuning** and **agent-orchestration** debugging, skip the `ResponsesHostServer` HTTP layer entirely. Build the agent in-process and call `run_async()` directly. ### Worked example `tests/local_smoke.py` (template ships in `references/local-stack/local_smoke.py`): ```python import asyncio import os from agent.container import build_agent # PoC's existing factory async def main(): os.environ.setdefault("MCP_SERVER_FQDN", "localhost:8000") os.environ.setdefault("MODEL_DEPLOYMENT_NAME", "gpt-5.4-mini") os.environ.setdefault("FOUNDRY_PROJECT_ENDPOINT",