| name | testing-course-samples |
| description | Use when asked to validate, test, smoke-test, or run the course's notebook and code samples against a live Microsoft Foundry / Azure OpenAI configuration. Covers environment setup (.env, az login, packages), the scripts/validate-notebooks.ps1 runner, interpreting PASS/FAIL results, and which lessons need extra resources (Azure AI Search, GitHub MCP, Foundry Local, Playwright). |
Testing the Course Samples
Validate that the lesson notebooks and code samples run against a live
Microsoft Foundry / Azure OpenAI setup. The repo ships a runner at
scripts/validate-notebooks.ps1 that
executes every Python notebook headlessly and prints a PASS/FAIL matrix.
When to use
- "Validate all the notebooks / samples against my Azure subscription."
- "Smoke-test the course after upgrading packages or changing models."
- "Which lessons still pass / fail live?"
Do not use this for the AI Smoke Test GitHub Action (that validates deployed
hosted agents — see tests/README.md). This skill
runs the notebooks locally.
Prerequisites (check first)
- Python 3.12+ with course deps:
python -m pip install -r requirements.txt
plus the executor: python -m pip install nbconvert ipykernel.
.env at the repo root (copy from .env.example) with at least:
AZURE_AI_PROJECT_ENDPOINT — Foundry project endpoint
(https://<account>.services.ai.azure.com/api/projects/<project>)AZURE_AI_MODEL_DEPLOYMENT_NAME — a non-deprecated deployment (e.g. gpt-5-mini)AZURE_OPENAI_ENDPOINT (https://<account>.openai.azure.com) and AZURE_OPENAI_DEPLOYMENT
for lessons that call Azure OpenAI directly (Lesson 06, 02-azure-openai, 14 handoff/human-loop).
az login completed — samples authenticate with AzureCliCredential (Entra ID, keyless).- Verify the model deployment exists:
az cognitiveservices account deployment list -g <rg> -n <account> -o table.
Running the validation
# All Python notebooks (skips .NET, .venv, site-packages, translations, skill assets)
pwsh scripts/validate-notebooks.ps1
# A single lesson, with a longer per-cell timeout
pwsh scripts/validate-notebooks.ps1 -Filter '08-*' -Timeout 600
# Just list what would run (no execution)
pwsh scripts/validate-notebooks.ps1 -List
# Explicit interpreter (if `python` is not on PATH, e.g. Windows Store alias)
pwsh scripts/validate-notebooks.ps1 -Python "C:/path/to/python.exe"
The script writes executed copies, per-notebook logs, and results.json to
$env:TEMP\aiab-nbval and exits with the number of failures.
Transient failures (shared-subscription HTTP 429 rate limits, an occasional
AzureCliCredential token hiccup, or a timeout) are retried automatically
(-Retries, default 2, with -RetryDelaySeconds backoff, default 20). If a
model deployment is regularly 429-ing, check the subscription's GlobalStandard
TPM quota (az cognitiveservices usage list -l <region>) — raising a single
deployment's capacity does not help when the subscription quota is exhausted.
Interpreting results
PASS — the notebook ran end-to-end with no cell error.FAIL — the first *Error / *Exception line is shown; open the matching
log_*.txt in the output dir for the full traceback.- A single notebook's failure is bounded by
-Timeout (per cell), so a hung
human-in-the-loop cell surfaces as StdinNotImplementedError rather than hanging.
Lessons that need extra resources (expected to fail without them)
| Lesson | Extra requirement |
|---|
| 05 Agentic RAG | Azure AI Search (AZURE_SEARCH_SERVICE_ENDPOINT, key) — has an in-memory fallback path |
| 11 MCP / GitHub | GitHub MCP server + PAT |
| 13 memory (cognee) | cognee configured with a model provider |
| 15 browser-use | Playwright browsers installed (playwright install) + AZURE_OPENAI_CHAT_DEPLOYMENT_NAME |
| 17 local agent | Foundry Local runtime + a downloaded Qwen model (on-device, no cloud) |
*-dotnet-* notebooks | .NET Interactive kernel (excluded by default; use -IncludeDotnet) |
Reporting back
Summarise as a PASS/FAIL table grouped by lesson. Separate genuine regressions
(code/config bugs to fix) from environment gaps (missing Search/Foundry Local/PAT),
and cite the failing log_*.txt for each real failure.
Disclaimer:
This document has been translated using AI translation service Co-op Translator. While we strive for accuracy, please be aware that automated translations may contain errors or inaccuracies. The original document in its native language should be considered the authoritative source. For critical information, professional human translation is recommended. We are not liable for any misunderstandings or misinterpretations arising from the use of this translation.
