testing-course-samples

name	testing-course-samples
description	Use when asked to validate, test, smoke-test, or run the course's notebook and code samples against a live Microsoft Foundry / Azure OpenAI configuration. Covers environment setup (.env, az login, packages), the scripts/validate-notebooks.ps1 runner, interpreting PASS/FAIL results, and which lessons need extra resources (Azure AI Search, GitHub MCP, Foundry Local, Playwright).

Testing the Course Samples

Validate that the lesson notebooks and code samples run against a live Microsoft Foundry / Azure OpenAI setup. The repo ships a runner at scripts/validate-notebooks.ps1 that executes every Python notebook headlessly and prints a PASS/FAIL matrix.

When to use

• "Validate all the notebooks / samples against my Azure subscription."
• "Smoke-test the course after upgrading packages or changing models."
• "Which lessons still pass / fail live?"

Do not use this for the AI Smoke Test GitHub Action (that validates deployed hosted agents — see tests/README.md). This skill runs the notebooks locally.

Prerequisites (check first)

1. Python 3.12+ with course deps: python -m pip install -r requirements.txt plus the executor: python -m pip install nbconvert ipykernel.
2. .env at the repo root (copy from .env.example) with at least:
   • AZURE_AI_PROJECT_ENDPOINT — Foundry project endpoint (https://<account>.services.ai.azure.com/api/projects/<project>)
   • AZURE_AI_MODEL_DEPLOYMENT_NAME — a non-deprecated deployment (e.g. gpt-5-mini)
   • AZURE_OPENAI_ENDPOINT (https://<account>.openai.azure.com) and AZURE_OPENAI_DEPLOYMENT for lessons that call Azure OpenAI directly (Lesson 06, 02-azure-openai, 14 handoff/human-loop).
3. az login completed — samples authenticate with AzureCliCredential (Entra ID, keyless).
4. Verify the model deployment exists: az cognitiveservices account deployment list -g <rg> -n <account> -o table.

Running the validation

# All Python notebooks (skips .NET, .venv, site-packages, translations, skill assets)
pwsh scripts/validate-notebooks.ps1

# A single lesson, with a longer per-cell timeout
pwsh scripts/validate-notebooks.ps1 -Filter '08-*' -Timeout 600

# Just list what would run (no execution)
pwsh scripts/validate-notebooks.ps1 -List

# Explicit interpreter (if `python` is not on PATH, e.g. Windows Store alias)
pwsh scripts/validate-notebooks.ps1 -Python "C:/path/to/python.exe"

The script writes executed copies, per-notebook logs, and results.json to $env:TEMP\aiab-nbval and exits with the number of failures.

Transient failures (shared-subscription HTTP 429 rate limits, an occasional AzureCliCredential token hiccup, or a timeout) are retried automatically (-Retries, default 2, with -RetryDelaySeconds backoff, default 20). If a model deployment is regularly 429-ing, check the subscription's GlobalStandard TPM quota (az cognitiveservices usage list -l <region>) — raising a single deployment's capacity does not help when the subscription quota is exhausted.

Interpreting results

• PASS — the notebook ran end-to-end with no cell error.
• FAIL — the first *Error / *Exception line is shown; open the matching log_*.txt in the output dir for the full traceback.
• A single notebook's failure is bounded by -Timeout (per cell), so a hung human-in-the-loop cell surfaces as StdinNotImplementedError rather than hanging.

Lessons that need extra resources (expected to fail without them)

Lesson	Extra requirement
05 Agentic RAG	Azure AI Search (AZURE_SEARCH_SERVICE_ENDPOINT, key) — has an in-memory fallback path
11 MCP / GitHub	GitHub MCP server + PAT
13 memory (cognee)	cognee configured with a model provider
15 browser-use	Playwright browsers installed (playwright install) + AZURE_OPENAI_CHAT_DEPLOYMENT_NAME
17 local agent	Foundry Local runtime + a downloaded Qwen model (on-device, no cloud)
*-dotnet-* notebooks	.NET Interactive kernel (excluded by default; use -IncludeDotnet)

Reporting back

Summarise as a PASS/FAIL table grouped by lesson. Separate genuine regressions (code/config bugs to fix) from environment gaps (missing Search/Foundry Local/PAT), and cite the failing log_*.txt for each real failure.