testing-course-samples

name	testing-course-samples
description	Use when asked to validate, test, smoke-test, or run the course's notebook and code samples against a live Microsoft Foundry / Azure OpenAI configuration. Covers environment setup (.env, az login, packages), the scripts/validate-notebooks.ps1 runner, interpreting PASS/FAIL results, and which lessons need extra resources (Azure AI Search, GitHub MCP, Foundry Local, Playwright).

Testing the Course Samples

Validate that the lesson notebooks and code samples run against a live Microsoft Foundry / Azure OpenAI setup. The repo ships a runner at scripts/validate-notebooks.ps1 that executes every Python notebook headlessly and prints a PASS/FAIL matrix.

When to use

• "Validate all the notebooks / samples against my Azure subscription."
• "Smoke-test the course after upgrading packages or changing models."
• "Which lessons still pass / fail live?"

Do not use this for the AI Smoke Test GitHub Action (that validates deployed hosted agents — see tests/README.md). This skill runs the notebooks locally.

Prerequisites (check first)

1. Python 3.12+ with course deps: python -m pip install -r requirements.txt plus the executor: python -m pip install nbconvert ipykernel.
2. .env at the repo root (copy from .env.example) with at least:
   • AZURE_AI_PROJECT_ENDPOINT — Foundry project endpoint (https://<account>.services.ai.azure.com/api/projects/<project>)
   • AZURE_AI_MODEL_DEPLOYMENT_NAME — a non-deprecated deployment (e.g. gpt-5-mini)
   • AZURE_OPENAI_ENDPOINT (https://<account>.openai.azure.com) and AZURE_OPENAI_DEPLOYMENT for lessons that call Azure OpenAI directly (Lesson 06, 02-azure-openai, 14 handoff/human-loop).
3. az login completed — samples authenticate with AzureCliCredential (Entra ID, keyless).
4. Verify the model deployment exists: az cognitiveservices account deployment list -g <rg> -n <account> -o table.

Running the validation

# All Python notebooks (skips .NET, .venv, site-packages, translations, skill assets)
pwsh scripts/validate-notebooks.ps1

# A single lesson, with a longer per-cell timeout
pwsh scripts/validate-notebooks.ps1 -Filter '08-*' -Timeout 600

# Just list what would run (no execution)
pwsh scripts/validate-notebooks.ps1 -List

# Explicit interpreter (if `python` is not on PATH, e.g. Windows Store alias)
pwsh scripts/validate-notebooks.ps1 -Python "C:/path/to/python.exe"

The script writes executed copies, per-notebook logs, and results.json to $env:TEMP\aiab-nbval and exits with the number of failures.

Transient failures (shared-subscription HTTP 429 rate limits, an occasional AzureCliCredential token hiccup, or a timeout) are retried automatically (-Retries, default 2, with -RetryDelaySeconds backoff, default 20). If a model deployment is regularly 429-ing, check the subscription's GlobalStandard TPM quota (az cognitiveservices usage list -l <region>) — raising a single deployment's capacity does not help when the subscription quota is exhausted.

Interpreting results

• PASS — the notebook ran end-to-end with no cell error.
• FAIL — the first *Error / *Exception line is shown; open the matching log_*.txt in the output dir for the full traceback.
• A single notebook's failure is bounded by -Timeout (per cell), so a hung human-in-the-loop cell surfaces as StdinNotImplementedError rather than hanging.

Lessons that need extra resources (expected to fail without them)

Lesson	Extra requirement
05 Agentic RAG	Azure AI Search (AZURE_SEARCH_SERVICE_ENDPOINT, key) — has an in-memory fallback path
11 MCP / GitHub	GitHub MCP server + PAT
13 memory (cognee)	cognee configured with a model provider
15 browser-use	Playwright browsers installed (playwright install) + AZURE_OPENAI_CHAT_DEPLOYMENT_NAME
17 local agent	Foundry Local runtime + a downloaded Qwen model (on-device, no cloud)
*-dotnet-* notebooks	.NET Interactive kernel (excluded by default; use -IncludeDotnet)

Reporting back

Summarise as a PASS/FAIL table grouped by lesson. Separate genuine regressions (code/config bugs to fix) from environment gaps (missing Search/Foundry Local/PAT), and cite the failing log_*.txt for each real failure.

---

Disclaimer: This document has been translated using AI translation service Co-op Translator. While we strive for accuracy, please be aware that automated translations may contain errors or inaccuracies. The original document in its native language should be considered the authoritative source. For critical information, professional human translation is recommended. We are not liable for any misunderstandings or misinterpretations arising from the use of this translation.