testing-course-samples

# Testing the Course Samples Validate that the lesson notebooks and code samples run against a live Microsoft Foundry / Azure OpenAI setup. The repo ships a runner at [`scripts/validate-notebooks.ps1`](../../../scripts/validate-notebooks.ps1) that executes every Python notebook headlessly and prints a PASS/FAIL matrix. ## When to use - "Validate all the notebooks / samples against my Azure subscription." - "Smoke-test the course after upgrading packages or changing models." - "Which lessons still pass / fail live?" Do **not** use this for the AI Smoke Test GitHub Action (that validates *deployed* hosted agents — see [`tests/README.md`](../../../tests/README.md)). This skill runs the notebooks locally. ## Prerequisites (check first) 1. **Python 3.12+** with course deps: `python -m pip install -r requirements.txt` plus the executor: `python -m pip install nbconvert ipykernel`. 2. **`.env` at the repo root** (copy from [`.env.example`](../../../.env.example)) with at least: - `AZURE_AI_PROJECT_ENDPOINT` — Foundry project endpoint (`https://<account>.services.ai.azure.com/api/projects/<project>`) - `AZURE_AI_MODEL_DEPLOYMENT_NAME` — a non-deprecated deployment (e.g. `gpt-5-mini`) - `AZURE_OPENAI_ENDPOINT` (`https://<account>.openai.azure.com`) and `AZURE_OPENAI_DEPLOYMENT` for lessons that call Azure OpenAI directly (Lesson 06, 02-azure-openai, 14 handoff/human-loop). 3. **`az login`** completed — samples authenticate with `AzureCliCredential` (Entra ID, keyless). 4. Verify the model deployment exists: `az cognitiveservices account deployment list -g <rg> -n <account> -o table`. ## Running the validation ```powershell # All Python notebooks (skips .NET, .venv, site-packages, translations, skill assets) pwsh scripts/validate-notebooks.ps1 # A single lesson, with a longer per-cell timeout pwsh scripts/validate-notebooks.ps1 -Filter '08-*' -Timeout 600 # Just list what would run (no execution) pwsh scripts/validate-notebooks.ps1 -List # Explicit interpreter (if `python` is not on PATH, e.g. Windows Store alias) pwsh scripts/validate-notebooks.ps1 -Python "C:/path/to/python.exe" ``` The script writes executed copies, per-notebook logs, and `results.json` to `$env:TEMP\aiab-nbval` and exits with the number of failures. Transient failures (shared-subscription HTTP 429 rate limits, an occasional `AzureCliCredential` token hiccup, or a timeout) are retried automatically (`-Retries`, default 2, with `-RetryDelaySeconds` backoff, default 20). If a model deployment is regularly 429-ing, check the subscription's GlobalStandard TPM quota (`az cognitiveservices usage list -l <region>`) — raising a single deployment's capacity does not help when the *subscription* quota is exhausted. ## Interpreting results - `PASS` — the notebook ran end-to-end with no cell error. - `FAIL` — the first `*Error` / `*Exception` line is shown; open the matching `log_*.txt` in the output dir for the full traceback. - A single notebook's failure is bounded by `-Timeout` (per cell), so a hung human-in-the-loop cell surfaces as `StdinNotImplementedError` rather than hanging. ## Lessons that need extra resources (expected to fail without them) | Lesson | Extra requirement | |--------|-------------------| | 05 Agentic RAG | Azure AI Search (`AZURE_SEARCH_SERVICE_ENDPOINT`, key) — has an in-memory fallback path | | 11 MCP / GitHub | GitHub MCP server + PAT | | 13 memory (cognee) | `cognee` configured with a model provider | | 15 browser-use | Playwright browsers installed (`playwright install`) + `AZURE_OPENAI_CHAT_DEPLOYMENT_NAME` | | 17 local agent | Foundry Local runtime + a downloaded Qwen model (on-device, no cloud) | | `*-dotnet-*` notebooks | .NET Interactive kernel (excluded by default; use `-IncludeDotnet`) | ## Reporting back Summarise as a PASS/FAIL table grouped by lesson. Separate genuine regressions (code/config bugs to fix) from environment gaps (missing Search/Foundry Local/PAT), and cite the failing `log_*.txt` for each real failure.